依托于GitLab持续集成基础配置和使用

GitLab 服务器: 192.168.1.250 = gitlab.weiyigeek.top
Gitlab-Runner 服务器: 192.168.1.3

#方式1: GitLab官方安装脚本
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.rpm.sh | sudo bash
yum install -y gitlab-ci-multi-runner
gitlab-ci-multi-runner register


#方式2:清华大学镜像站源进行安装Gitlab-Runner
cat > /etc/yum.repos.d/gitlab-runner.repo <<END
[gitlab-runner]
name=gitlab-runner
baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-runner/yum/el7
repo_gpgcheck=0
gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/gpg.key
END
sudo yum makecache
sudo yum list gitlab-runner --showduplicates #查看可安装版本
sudo yum install gitlab-runner -y
# 此处您可以安装指定版本 其中 12.3.5 即为指定的版本号
# yum install gitlab-runner-12.3.5-1.x86_64 -y

[root@Gitlab-Runner ~]$gitlab-runner register
Runtime platform    arch=amd64 os=linux pid=20709 revision=4c96e5ad version=12.9.0
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://gitlab.weiyigeek.top/   #Gitlab地址
Please enter the gitlab-ci token for this runner:
qupxfdPtuzCckymoSCUu           #gitlab-ci令牌
Please enter the gitlab-ci description for this runner:
[initiator]: TestRunner       #Runner 描述
Please enter the gitlab-ci tags for this runner (comma separated):
TestRunner   #Runner 标签
Registering runner... succeeded  runner=qupxfdPt #表示注册成功
Please enter the executor: parallels, custom, docker, docker-ssh, docker+machine, docker-ssh+machine, kubernetes, shell, ssh, virtualbox:
shell #执行方式
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

$nano /etc/gitlab-runner/config.toml

#限制了整个GitLab Runner能并发处理job的数量
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "TestRunner"
  url = "http://gitlab.weiyigeek.top/"
  #注册生成的Runner 令牌
  token = "5UmJ5uECw5k6H_phzeiW"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "DockerRunner"
  url = "http://gitlab.weiyigeek.top/"
  token = "HrpstQA6eezWv5-9m5-d"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    # 在这里需要添加上 harbor 的地址，才能允许 pull 私有 registry 的镜像
    allowed_images = ["192.168.1.1/*:*"]
    tls_verify = false
    image = "alpine-node:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

## 1.注册runner 会以引导的方式询问相关参数的设置-交互式
$gitlab-runner register

## 2.查看已经注册的runner
$gitlab-runner list
Runtime platform            arch=amd64 os=linux pid=81864 revision=4c96e5ad version=12.9.0
Listing configured runners    ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml

# 3.删除gitlab-runner相关配置
sudo gitlab-runner uninstall

# 4.安装并设置--user(例如设置为root)
sudo gitlab-runner install --working-directory /home/gitlab-runner --user root 

# 5.临时采用root权限运行runner(坑多，在Gitlab-CI中由于采用的gitlab-runner用户常常操作文件时候提醒权限不足)
sudo gitlab-runner run

#6.将GitLab Runner安装为系统服务：
sudo gitlab-runner install -n "<service-name>" -u <user-name>

#7.启动服务
sudo gitlab-runner start -n "<service-name>"

#Path: secopsdev/.gitlab-ci.yml
#使用.gitlab-ci.yml配置你的项目,注意下面Tags是必须要指定的否则CI报错找不到Runner;
stages:
  - test
  - build
  - deploy
#每次
before_script:
  - echo "#Before script section" >> WeiyiGeek.txt
  - echo "For example you might run an update here or install a build dependency"
  - echo "Or perhaps you might print out some debugging details"

after_script:
  - echo "#After script section" >> WeiyiGeek.txt
  - echo "For example you might do some cleanup here"

test1:
  stage: test
  script:
    - echo "(1) Do a test here - 正在进行代码测试 阶段1 "
    - echo "For example run a test suite"
  tags:
    - TestRunner
    
test2:
  stage: test
  script:
    - echo "(2) Do another parallel test here - 正在进行代码测试 阶段2 并行测试 "
    - echo "For example run a lint test"
  tags:
    - TestRunner
    
build:
  stage: build
  script:
    - echo "(3) Building the app - 正在对项目进行编译 "
  tags:
    - TestRunner
    
deploy:
  stage: deploy
  script:
    - echo "(4) Deploy to staging server :" $url
    - echo "(4) Deploy to staging server :" $(env)
  environment:
    name: staging
    url: https://staging.weiyigeek.top
  only:
    - master
  tags:
    - TestRunner

Please enter the executor: parallels, custom, docker, docker-ssh, docker+machine, docker-ssh+machine, kubernetes, shell, ssh, virtualbox:

#Repo
/home/gitlab-runner/builds/5UmJ5uEC/0/WeiyiGeek/blog

#Cache
/home/gitlab-runner/cache/WeiyiGeek/blog

[runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    # 在这里需要添加上 harbor 的地址，才能允许 pull 私有 registry 的镜像
    allowed_images = ["192.168.1.1/*:*"]
    tls_verify = false
    image = "alpine-node:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

$ sudo docker images
REPOSITORY                          TAG
gitlab/gitlab-runner-helper     x86_64-3afdaba6
gitlab/gitlab-runner-helper     x86_64-cf91d5e1

volumes = ["/home/x1twbm/.ssh:/root/.ssh:ro"]

/home/gitlab-runner/cache/gitlab用户/项目名称/cache.zip
/home/gitlab-runner/cache/WeiyiGeek/blog/cache.zip

#全局缓存 global_cache
cache: &global_cache
  key: ${CI_JOB_NAME} #结合 GitLab CI/CD 中预定义的参数有不同的效果
  # 缓存 node_modules/目录 下次构建不会删除
  paths:  
    - node_modules/
  policy: pull-push #默认对缓存文件进行拉取与更新

#局部缓存 local_cache
local_cache:
  stage: build
  cache:   #如果在 job 内部也定义了 cache 配置，全局的配置就会被覆盖
    key: ${CI_JOB_NAME}
    policy: pull  #使用了pull 策略不会在 job 结束后进行文件的上传操作
    paths:
      - node_modules/ 
      - binaries/*.apk  #在binaries 目录下以 .apk 结尾的所有文件以及 .config 文件会被缓存下来。
  
#Cache 的继承
inherit_cache:
 cache:
    # 继承全局配置
    <<: *global_cache
    # 仅覆盖 cache:policy 的配置
    policy: pull


# Cache 的禁用
disable_cache
  cache: {}

[[runners]]
  limit = 10
  executor = "docker"
  [runners.cache]
    Type = "s3"
    Path = "path/to/prefix"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.example.com"
      AccessKey = "access-key"
      SecretKey = "secret-key"
      BucketName = "runner"
      Insecure = false

stages:
   - test1
   - test2
   - test3

cache:
   key: "$CI_COMMIT_REF_SLUG"
   paths:
     - Test.txt

job1:
   stage: test1
   script:
     - cat Test.txt
     - echo "aaaaaaaaaa" > Test.txt
     - cat Test.txt
   tags:
     - base-runner

job2:
   stage: test2
   cache:
     key: "$CI_COMMIT_REF_SLUG"
     paths:
       - Test.txt
     policy: pull  #关注点
   script:
     - cat Test.txt
     - echo "bbbbbbbbbbb" >> Test.txt
     - cat Test.txt
   tags:
     - base-runner

job3:
   stage: test3
   script:
     - ls
     - cat Test.txt
     - echo "cccccccc" >> Test.txt
     - cat Test.txt
   tags:
     - base-runner

#job1 结果
Checking cache for test-wer-1...
#首次检查缓存未存在进行创建缓存
$ echo "aaaaaaaaaa" > Test.txt
$ cat Test.txt
aaaaaaaaaa
Creating cache test-wer-1...
Test.txt: found 1 matching files                                  
Uploading cache.zip to http://192.168.12.139:9000/runner/cache-path/project/1242/test-wer-1   #上传 cache 路径
Created cache
Job succeeded


#job2 结果
Checking cache for test-wer-1...
cache.zip is up to date  #缓存进行了下载              
Successfully extracted cache
$ cat Test.txt
aaaaaaaaaa
$ echo "bbbbbbbbbb" >> Test.txt
$ cat Test.txt
aaaaaaaaaa
bbb
Not uploading cache test-wer-1 due to policy # pull策略不上传缓存
Job succeeded


#job3 结果
Checking cache for test-wer-1...
cache.zip is up to date                            
Successfully extracted cache
$ cat Test.txt
aaaaaaaaaa
$ echo "ccccccccc" >> Test.txt
$ cat Test.txt
aaaaaaaaaa
ccccccccc
Creating cache test-wer-1...
Test.txt: found 1 matching files                
22.txt: found 1 matching files                     
Uploading cache.zip to http://192.168.12.139:9000/runner/cache-path/project/1242/test-wer-1  #上传 cache 路径
Created cache
Job succeeded

ERROR: Registering runner... failed runner=qRGh2M86 status=500 Internal Server Error
PANIC: Failed to register this runner. Perhaps you are having network problems

#此处以第一种方式为列
使用 root 用户从 web 端登录到 gitlab 管理中心 http://${ip}/admin 。管理中心 --> 设置 --> 网络 –> 外发请求
#设置如下
* 勾选 允许Webhook和服务对本地网络的请求
* 勾选 允许系统钩子向本地网络发送的请求

==> /var/log/gitlab/gitlab-rails/production.log <==
Started PATCH "/admin/application_settings/network" for 10.0.30.2 at 2020-03-10 11:08:20 +0000
Processing by Admin::ApplicationSettingsController#network as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "application_setting"=>
{"allow_local_requests_from_web_hooks_and_services"=>"[FILTERED]",
"allow_local_requests_from_system_hooks"=>"[FILTERED]", "outbound_local_requests_whitelist_raw"=>"",
"dns_rebinding_protection_enabled"=>"1"}}
Completed 500 Internal Server Error in 40ms (ActiveRecord: 14.5ms | Elasticsearch: 0.0ms)
OpenSSL::Cipher::CipherError ():
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"PATCH","path":"/admin/application_settings/network","format":"html","controller":"Admin::
ApplicationSettingsController","action":"network","status":500,"error":"OpenSSL::Cipher::CipherError
: ","duration":40.08,"view":0.0,"db":14.47,"time":"2020-03-10T11:08:20.061Z","params":
[{"key":"utf8","value":"✓"},{"key":"_method","value":"patch"},{"key":"authenticity_token","value":"
[FILTERED]"},{"key":"application_setting","value":
{"allow_local_requests_from_web_hooks_and_services":"
[FILTERED]","allow_local_requests_from_system_hooks":"
[FILTERED]","outbound_local_requests_whitelist_raw":"","dns_rebinding_protection_enabled":"1"}}],"re
mote_ip":"10.0.30.2","user_id":1,"username":"root","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87
Safari/537.36","queue_duration":2.55,"correlation_id":"9rvflpksdj2"}

#命令行输入 gitlab-rails console ，然后输入ApplicationSetting.current reset_runners_registration_token! 即可，这样在保存修改的时候就不会再报 500的问题了。
╭─[email protected] ~
╰─# gitlab-rails console
--------------------------------------------------------------------------------
GitLab: 12.3.5 (2417d5becc7)
GitLab Shell: 10.0.0
PostgreSQL: 10.9
--------------------------------------------------------------------------------
Loading production environment (Rails 5.2.3)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
irb(main):002:0> exit

build:
  stage: build
  script: echo "Building the app"
  #告诉 Runner 我们应用哪个标签
  tags:
    - TestRunner

#1.将gitlab-runner用户加入docker组
$usermod -aG docker gitlab-runner

#2.gitlab-runner加入的组出现docker即没问题
$groups gitlab-runner
gitlab-runner : gitlab-runner docker

#因runner运行时的执行者是gitlab-runner账户，需要在gitlab-runner账号下设置访问https类网站时，免验证
[[email protected] ~]# su - gitlab-runner
[[email protected] ~]$ git config --global http."sslVerify" false
#查看
[[email protected] ~]$ cat /home/gitlab-runner/.gitconfig 
[http]
  sslVerify = false

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
time="2020-04-17T11:12:33Z" level=error msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix

如何实现与gitlab互联

基于 Docker 的 CI/CD 工具 Drone 所有编译、测试的流程都在 Docker 容器中进行。

开发者只需在项目中包含 .drone.yml 文件，将代码推送到 git 仓库，Drone 就能够自动化的进行编译、测试、发布。

本文仅记录 Drone是如何跟Gitlab打通的过程。

由于Drone是基础Docker方式运行，所以需要有一个Docker环境。关于如何安装部署Docker这里就不细说了。

前要说明
由于drone 1.x版本跟0.8版本的变化也挻多，有些旧的参数已经弃用或者变更了，详细情况要参考官方文档。https://docs.drone.io/

1、配置gitlab applications:

登陆到Gitlab系统点击个人头像—-Settings—-Applications添加一个应用:
这里是一个变化，0.8版Redirect URL后面是跟：authorize ，而1.X版本使用：login

提交后就会自动生成以下ID和Key：

2、首先要拉取一个drone镜像：
通过docker search drone 可以看到很多相关的镜像，这里拉取第一个官方的镜像，大约63M左右，因为官方镜像也已经更新到了1.X版本，所以直接docker pull drone/drone就是1.X版的。

[[email protected] ~]# docker pull drone/drone
[[email protected] ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
helloworld          latest              232ac0d24f85        21 hours ago        6.96MB
plugins/base        multiarch           92bf59f52f26        23 hours ago        293kB
drone/drone         latest              16646fab2afb        4 days ago          63.2MB

3、启动Drone服务

以下配置直接使用官网给出的示例；可以看到1.X版本跟之前的0.8版本有些地方，参数也有变化，以下这个配置是单机版的，就是server和agnet都整合运行在同一机器上，这个适合测试环境，或者使用量比较小的环境下。

docker run 
  --volume=/var/run/docker.sock:/var/run/docker.sock 
  --volume=/var/lib/drone:/data 
  --env=DRONE_LOGS_DEBUG=true 
  --env=DRONE_GIT_ALWAYS_AUTH=false 
  --env=DRONE_GITLAB_SERVER=http://10.0.10.8 
  --env=DRONE_GITLAB_CLIENT_ID=09cb3f3bbdc607060f1fdb71b51e4a9e4db4c576955763a1a4efec078edf2d6b 
  --env=DRONE_GITLAB_CLIENT_SECRET=8360a4e0d5697127f3e8a77440e884c3b0b93375c563cfaa407074d8b231c159 
  --env=DRONE_RUNNER_CAPACITY=3 
  --env=DRONE_SERVER_HOST=10.0.10.68 
  --env=DRONE_SERVER_PROTO=http 
  --env=DRONE_TLS_AUTOCERT=false 
  --publish=80:80 
  --publish=443:443 
  --restart=always 
  --detach=true 
  --name=drone  
  drone/drone:latest

说明：

    DRONE_GITLAB_SERVER ：修改自己的gitlab仓库地址
    DRONE_GITLAB_CLIENT_ID : 这个是在Gitlab里边配置的可参考上图
    DRONE_GITLAB_CLIENT_SECRET : 同上当在gitlab中配置了一个applications就会自动生成ID，和SECRET
    DRONE_SERVER_HOST ：drone服务运行的Ip地址
    DRONE_SERVER_PROTO ：走哪种协议，这里直接使用http 即可，如果是外网能够访问的建议https.
    DRONE_LOGS_DEBUG=true :开启Debug日志

当启动完Drone容器后再打开浏览器访问：http://10.0.10.68就会跳转到Gitlab页面,然后会提示授权。

授权完成后又会回到drone的界面了，如果Gitlab上有项目的就会把项目的信息同步过来。

至此，drone就可以实现Gitlab的连通了。

特别提醒

错误提示：

Login Failed. invalid character '<' looking for beginning of value

使用1.x版本的drone配置旧版本的gitlab(8.8.X)时遇到了一个错误，最后更新了gitlab版本到11.X后才能正常回调到drone的登陆页面。