CI/CD Flow

Gitlab's vision

GitLab's vision covers the entire flow from idea to production. Many of these pieces will be provided directly by GitLab, others will be provided by bundling third-party solutions. We don't want to own or manage the production infrastructure, but we do want to help developers coordinate getting their code into production; providing convenience and confidence to the developer in an integrated way.

Gitlab's vision 涵蓋整個從發想到產品的流程. 每一個流程片段Gitlab都能提供相對的服務，就算目前還有空缺的，也將能透過整合第三方軟體來解決．我們並不想要擁有與建立產品的基礎建設，而是要幫助開發者能夠讓他們的code有標準的放入他們的產品，提供更方便可信任的方法去整合他們的工作 Scope of Gitlab's vision

The CI/CD vision focuses on steps 6 through 9: Test (CI), part of Review (MR), Staging (CD), and part of Production (Chatops). When viewed through the CI/CD lens, we can group the scope into CI, CD, and things that are currently beyond any definition of CD.

CI/CD關注在第6-9個步驟（Idea是第一步驟)：自動化測試(持續整合,CI),檢驗(code review),持續建置(CD),ChatOps(可以透過Chatbot了解產品狀況). 透過CI/CD, 我們做的超越現有任何CD的範疇．

CI/CD flowchart

One obvious question is, what's the difference between Deploy and Deliver? I'm a big believer in decoupling deployment of code from delivery of a feature, mostly using feature flags. Continuous integration helps improve the speed of development, but feature flags takes it to another level, giving you the confidence to integrate code even more often while providing a gradual and granular method for delivery. 最常見的問題是，這有什麼不同於傳統的Deploy(建置)與Deliver(發布),我們認為應該將建置流程從發布流程解構出來. 持續整合加速我們的開發速度, 但加入發佈狀態旗號, 將讓我們在不斷的快速的建置流程中，更有信心地去整合我們的程式碼．

程式開發整合管線化

There are three types of pipelines we care about that often use the single shorthand of "pipeline". We often talk about them as if each one is "the" pipeline, but really, they're just pieces of a single, comprehensive pipeline.

這裡有三種管線話化流程, 我們通常稱之為 管件化. 但事實上這都是整個程式開發整合的一部份而已 dev flow

  • CI Pipeline: Build and test stages defined in .gitlab-ci.yml

CI 管線: 透過.gitlab-ci.yml 裡面的 build 與 test 階段可以幫忙

  • Deploy Pipeline: Deploy stage(s) defined in .gitlab-ci.yml The flow of deploying code to servers through various stages: e.g. development to staging to production

建置管線: 透過.gitlab-ci.yml 裡面各式各樣的 staging 去協助 Deploy

  • Project Pipeline: Cross-project CI dependencies, particularly for micro-services, but also for complicated build dependencies: e.g. api -> front-end, ce/ee -> omnibus.

專案管線: 跨專案的CI依賴, 特別是針對微服務或者是複雜的建置依賴, 例如: api 對於 前端, ce/ee(??)對於 omnibus

Development workflows to accommodate:

  • Branch Flow (e.g. different branch for dev, qa, staging, production)
  • Trunk-based Flow (e.g. feature branches and single master branch, possibly with tags for releases)
  • Fork-based Flow (e.g. merge requests come from forks)

開發流程包含:

  • 分支流程(針對不同目的的開發分支, 開發dev, 品管qa, 建置(staging), 部署(production)
  • 主幹流程(有多特色分支，但只會有一隻主線與上Tags做發布)
  • Fork流程(整併從其他fork出去的requests)

Example flow:

pipelines goal

Issues to support pipelines overall

  1. CI pipeline for a single commit, single project DONE
  2. Run CI/CD on Merge Requests, not just branches, especially for fork-based flows

  3. Deploy pipelines

    1. Manual steps (e.g. deploy same SHA from staging to production) DONE
    2. Cross-commit (e.g. before and after a merge)
    3. Link between related commits, merge commits, and tags
    4. Show status of merge request beyond merge. (e.g. add staging and production deploys to MR activity stream)

  4. Multi-project pipelines

  5. First-class triggers
  6. [Cross-project dependencies](https://gitlab.com/gitlab-org/gitlab-ce/issues/17069
  7. Cross-project artifacts
  8. Link between project pipeline views
  9. Consolidated view of entire pipeline across projects
  10. Use Docker image registry and Docker Compose to run cross-project integration tests within single project's pipeline

與管線開發相關的議題一覽表

  1. CI pipeline 如何在一個專案上進行
  2. 執行CI/CD 在整併Request的時候

  3. CD pipeline相關

    1. 手動流程讓建置上發布(確定是相同的SHA上產品)
    2. 整併前後的commit ?!
    3. 連結相關的commit 與整併commit 還有tags
    4. 呈現merge後的狀態

  4. 多專案管線

  5. ??
  6. 跨專案相依
  7. ??
  8. 連結專案管線
  9. 連結跨專案管線一致化
  10. 用Docker image registry 與 Docker Compose 跑跨專案整合測試, 在單一專案管線上

Staging 管線階段

Build

GitLab CI provides an explicit build stage already and the concept of build artifacts. As we expand to a complete CD solution, we might need to separate out build artifacts or "releases" from test artifacts. For example, you might want your test runner to create a JUnit-style output file which is available for external consumption, but not included in the build image sent to production. Creation of an explicit build aligns well with Docker where the result of the build stage is a Docker image which is stored in a registry and later pulled for testing and deployment.

GitLab CI 提供明確的build 階段與建置檔案. 當我們延伸完整的CD流程, 我們就也許需要將build檔案與test用檔案在為了releases階段而分開來使用. 舉例來說, 你會需要在test階段使用JUnit檔案, 但並不需要建立一個docker image 給其他人使用. 透過一個顯而易見的build stage, 讓建置好的docker image 提供給 test 或者是 deploy使用

Builds as first-class citizen (aka build artifacts):

  1. Build history (of artifacts), a view of releases
  2. Deploy specific build to specific environment
  3. Rollback to previous build
  4. Docker images (storage, download, external usage, deployment, use in cross-project testing) DONE

建置是最重要的階段

  1. 建置歷史
  2. 用特別的環境建置
  3. 回到之前的版本
  4. 使用Docker image(為了儲存,下載, 外部使用, 發布, 跨專案測試)

Test

  1. Integration with third-party services like CodeClimate
  2. Report more than just pass/fail, report improving, degrading, above/below threshold of change
  3. Detect unnecessary builds/tests and skip them (e.g. merge of a MR off master/head where no files have changed)
  4. Auto-parallelize tests, splitting across files or even individual tests
  5. Load-balance tests so that each run will take roughly equal time, resulting in shortest wall-clock time
  6. Provides examples and/or wizard to get started
  7. Process JUnit-style output
  1. 整合第三方服務
  2. 回報更多不同面向的資訊
  3. 偵測不必要的build與test然後略過這西
  4. 自動平行化測試, 分開檔案與測試流程
  5. 平衡負載測試，讓測試時間變短
  6. 提供案例示範如何開始
  7. JUnit style 的輸出

Deploy

A key part of CD is being able to deploy. We currently have this ability via the dpl gem and the deploy stage in .gitlab-ci.yml. We need to go further.

CD最關鍵的一步是發布, 現我們有能力透過dpl與deploy階段在.gitlab-ci.yml來進行.

  1. Compute Platform config (e.g. add DO or AWS creds to project/group)
  2. Other services (e.g. TestFlight, Apple)

  3. Deploys as first-class entry

    1. New command rather than dpl gem
    2. Deploy history view DONE
    3. Rollback to previous deploy DONE
  4. Environments as first-class entry DONE
    1. List of environments DONE
    2. Current state and history of environments DONE
  5. Releases as first-class entry?
    1. Release = build + config, so this implies we'd manage config
  6. Deploy/Promote without rebuilding DONE
  7. Review Apps (e.g. create ephemeral apps on compute platform for each merge request)
  8. Support multiple rollout strategies
    1. Incremental deploys (make one, break one)
    2. Canary deploys (deploy to small % of production servers)
    3. Blue/green deploys (deploy to alternating servers, switch routing)
    4. Traffic vectoring (route % of production traffic to server running new release)

  9. Runner plugins for extensible deploy syntax in .gitlab-ci.yml

  10. 計算平台的設定

  11. 其他服務
  12. Deploy 應該是最高等級項目
    1. 新指令超越dpl??
    2. deploy 歷史
    3. 回到上一版deploy
  13. 環境當作最高等級項目
    1. 列出所有的環境
    2. 現在環境與環境的歷史
  14. 發布是第一級項目
    1. Release = build + config
  15. Deploy 不需要重新build
  16. Review Apps
  17. 支援多種首發策略
  18. 額外的擴充延伸套件提供給.gitlab-ci.yml

Deliver

Developer tools have an opportunity not only to provide great functionality, but to promote best practices. One such best practice is to put new features through ever-increasing exposure.

開發者工具不只提供很棒的功能, 也提供許多很棒的案例．最好的案例是透過不斷新功能曝光．

Feature flags allow developers to decouple deployment from delivery. Code can be put into production while the feature is still turned off. Features can then be rolled out to internal users, beta users, a percentage of production users, and then to everyone in ever-increases spheres. Not only does this help facilitate healthy real-world testing, it helps stay true to the premise of continuous integration which is that everything is checked into master often and deployed. This reduces risk.

Feature flags 讓 開發者能夠將 deploy 從 delivery分開. 程式碼可以放入產品之中, feature被關閉. Feature 可以揭露給不同階段的的人. 不只是幫助真實世界的測試, 也能確保每次deployed之時，都能如同CI所確定的, 這可以降低風險.

Developers can do this today themselves without our help. But let's not just support this flow, but encourage it, and add value to it. I imagine the Features list to not just show me an on/off for each feature, but which stage that feature is in. e.g. off/alpha/beta/ga. And when moving a feature from one stage to another, I imagine something that feels like I’m moving it to the next stage. e.g. Showing a grid of groups that I can enable the feature for, with the groups sorted by stage (or size as a proxy).

現在開發者已經可以靠自己完成不需要我們的幫忙, 但不只是提供這樣的flow,還要鼓勵他們，讓他更有價值. 我想像Features list 不只是讓我開關feature而已(?!?!)

I’d like to visually see, at a glance, which features are in private beta, for example. If a product manager tries to turn a feature on for GA, they’ll be non-invasively reminded that they should consider going through private beta first. Heck, even the choice of rollout percentages could be handled this way. e.g. 1%, 5%, 25%, 100% show up as selectable items in columns, giving a reminder of the company policy on rollouts.

我想要去視覺化那些在private beta的features.如果PM想要去啟用某個特定的feature為了GA, 他們會被提醒，他們應該思考先發布private beta優先.甚至做推出比例的選擇可以用這套流程，根據公司的推出(rollout)政策來處理．

If done right, it’ll encourage a good rollout process, and make it trivially easy for users to follow the flow, every time.

如果做得很好，他將會引出更好的rollout流程，與更容易讓使用者簡單好懂.

My ideal rollout is something like:

Dev team that created it -> internal alpha (all employees minus those giving customer-facing demos) -> select beta (~200 people that signed up for this specific feature) -> private beta (~1000 people that signed up for general beta) -> public beta -> 10% rollout -> GA

我的理想推出流程應該是

  1. Dev team 寫好
  2. internal alpha (所有的同仁 - 會給顧客demo)
  3. select beta (約200人來測試特定功能)
  4. private beta (約1000人來測試)
  5. public beta (開放測試)
  6. 10% rollout (上線)

  7. GA??? user-flags

  8. Integrate Feature Flags in an extensible way (so people can implement it however they want, but use a consistent interface for administration)

  9. Create, view, and toggle feature flags in each environment
  10. Display diffs between environments
  11. Tie feature flags related to merge requests

  12. On merge request view, show events for toggles of related feature flags in each environment

  13. 每個人都可以提供需要的Feature, 但只有管理者可以管理這些Feature

  14. 創造, 觀看, 觸發 feature flags在每個環境
  15. 顯示環境的差異
  16. 綁定Feature 與 Merge
  17. 在Merge request, 列出不同環境下,會綁定的feature 與事件

Monitor

  • Integrate with third-party services like Datadog for monitoring impact on system performance impact (e.g. CPU, memory)
  • Integrate with third-party services like Mixpanel for monitoring business impact (e.g. signups, conversion, revenue)
  • Watch metrics before and after a deploy to determine if a deploy causes system problems or negatively affects critical business metrics (e.g. breaks signup)
  • Automatically roll back changes that degrade too much
  • If following GitHub flow, block merges that degrade too much
  • Watch metrics for expected positive results
  • Tie metrics back to causal changes (e.g. commits and merge requests included in deploy)

  • Display metric impact on merge requests (even after merge) and feature flags

  • 整合第三方服務,例如DataDog去監控對於系統效能的衝擊(CPU, memory)

  • 整合第三方服務,例如Mixpanel去監控對於業務的影響(登入數,轉換率,報酬)
  • 監控對於正面結果的預期指標
  • 讓每次更新都跟這些指標連結
  • 呈現每次merge對於這些指標的衝擊

Misc 雜項

Distributed Systems 分散式系統

From 12 Factor, if there are multiple codebases, it’s not an app – it’s a distributed system. We should support distributed systems, especially mindful of the rising trend in microservices and Docker adoption. We've already covered multi-project pipelines above, but we can go further with system-level views and coordinated deploys.

根據12factor指出, 如果是多個codebases系統，那就不是一個app, 而是一個分散式系統. 我們應該要能支援分散式系統, 特別是微服務和Docker的趨勢. 我們已經可以涵蓋多專案pipelines的專案, 但我們要更進一步的從system等級的角度與整合deploy

  • 用看新角度 - 分散式系統- 來看一堆apps的整合 => 從專案管線化來看
  • 阻擋deploy 如果他依賴另外一個尚未deploy的版本, 例如:
    • 標註每個Merge requests
    • 追蹤Merge
    • 直到所有upstream changes都被整併結束後，才進行Merge requests
    • 直到所有upstream changes都被deploy才進行deploy
    • 這可以跨專案運作，因此每個服務都可以按照正確的順序進行
  • 按照順序deploy 所有有相關的apps到新環境, 例如自動化生產雲端系統, 所以我們會知道這個專案與其他是怎麼相關聯的.舉裡來說
  • New view of related apps in a distributed system => zoom out in the project pipeline
  • Block deploys of one component if it depends on a version of another component that has not been deployed yet to the same environment e.g.:
    • Annotate relationship between merge requests so that we know if one MR blocks another.
    • Track deploys of MRs to each environment so we know, for example, when a given MR is in production.
    • We can block merges of one MR until upstream changes are merged.
    • We can block deploys of one MR until upstream changes are deployed (to the same environment).
    • This can work across projects so individual services get deployed in the right order.
  • Coordinated deploy of all related apps to a new environment. e.g. autogenerate a "cloudformation" because we know how the projects relate to each other. This could, for example, be used by GitHost to spin up a new single-tenant instances of GitLab for a new customer.

Example 範例

A slightly more complete rendering of an example pipeline:

更完整個pipeline流程

cd

results matching ""

    No results matching ""