• notice
  • Congratulations on the launch of the Sought Tech site

How to build android source code repo warehouse

If your development is based on AOSP source code to build a warehouse, it is very necessary to build a repo server and deploy your own repo warehouse.

In reality, many companies manage the AOSP source code directly in a git repository, and there is nothing wrong with doing so. However, if the development team has a large number of people, there will be some obvious disadvantages:

  • The general size of the AOSP source code is 20GB, and it reaches more than 40GB after compilation. Each sync is time consuming.

  • The number of AOSP source code files is nearly 70W. The file index of a single warehouse will be very large, and each retrieval is very time-consuming.

  • The original code repositories of AOSP and chip platform manufacturers (such as qcom, mtk, rockchip, etc.) are generally provided in a repo repository structure, rather than a single git repository. After changing the warehouse structure, the workload of manual maintenance is relatively large.

The AOSP source code structure is complex and huge. In addition to its own code modules, it also integrates five or six hundred third-party warehouses. These warehouses will also be constantly updated and iterated. AOSP needs to maintain enough flexibility to maintain the entire code repository framework, so repo scripts are introduced to manage git repositories in batches. The number of git repositories for AOSP source code segmentation is as many as six or seven hundred.

If we want to keep the same warehouse structure as the upstream platform warehouse, it is very convenient and quick to synchronize the code, and we can flexibly manage the local extended warehouse, we need to build and deploy our own repo warehouse.

In actual team development practice, the code management framework is generally a combination of gerrit+repo. The installation and deployment and rights management of gerrit are not the content of this document. Readers are requested to search for online materials to install and deploy. The main contents of this document include:

  • Describes the management strategy of the repo repository.

  • How to build inventory warehouse.

  • How to batch shard git repositories.

  • Operational use of the repo repository.

The server operating system for this document is ubuntu18.04.

1 How does the repo manage the warehouse?

1.1 How does the repo work?

The new operating system installation package already contains the repo command program, which can be installed directly through the following command:

sudo apt-get install repo
# View the repo version
repo --version
<repo not installed>
repo launcher version 2.17
       (from /usr/local/bin/repo)
git 2.34.1
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0]
OS Linux 5.15.0-39-generic (#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022)
CPU x86_64 (x86_64)
Bug reports: https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue

After the installation is complete, we can see that the installation is actually a repo bootloader. It's not yet a complete management tool.

Take a look at the repo file through the cat command, which is actually a python3 script.

which repo
/usr/local/bin/repo
cat /usr/local/bin/repo
........

Several command sequences commonly used by the client when synchronizing the warehouse:

# Initialize repository
repo init -u ....
# checkout code
repo sync -j16
# create a local branch
repo start <BRANCH> --all
# Batch execute git commands
repo forall -c `GIT COMMAND`

Let's take the AOSP code of the official website of Tsinghua University as an example to operate the warehouse synchronization of the repo.

# Set up REPO_URL.
export REPO_URL="https://mirrors.tuna.tsinghua.edu.cn/git/git-repo/"

mkdir -p ~/aosp
cd aosp

# Initialize repository
repo init -u https://mirrors.tuna.tsinghua.edu.cn/git/AOSP/platform/manifest
Downloading Repo source from https://mirrors.tuna.tsinghua.edu.cn/git/git-repo/
remote: Enumerating objects: 7372, done.
remote: Counting objects: 100% (7372/7372), done.
remote: Compressing objects: 100% (3935/3935), done.
remote: Total 7372 (delta 4765), reused 5577 (delta 3363)
Receiving objects: 100% (7372/7372), 3.27 MiB | 3.58 MiB/s, done.
Processing delta: 100% (4765/4765), done.

... A new version of repo (2.21) is available.
... You should upgrade soon:
   cp /home/xxx/Documents/aosp/.repo/repo/repo /usr/local/bin/repo

Downloading manifest from https://mirrors.tuna.tsinghua.edu.cn/git/AOSP/platform/manifest
remote: Enumerating objects: 98793, done.
remote: Counting objects: 100% (98793/98793), done.
remote: Compressing objects: 100% (39423/39423), done.
remote: Total 98793 (delta 38918), reused 98123 (delta 38532)
In receiving objects: 100% (98793/98793), 25.23 MiB | 2.40 MiB/s, done.
Processing delta: 100% (38918/38918), done.

Your identity is: xxx <[email protected]>
If you want to change this, please re-run "repo init" with --config-name

Testing colorized output (for "repo diff", "repo status"):
 black red green yellow blue magenta cyan white
 bold dim ul reverse
Enable color display in this user account (y/N)? y

repo has been initialized in /home/xxx/Documents/aosp/
If this is not the directory in which you want to initialize repo, please run:
  rm -r /home/yl/Documents/aosp//.repo
and try again.

After the repo init command is executed, it actually downloads a .repo directory (the version of the repo command is outdated, you can update it according to the prompt, or you can not update it).

Let's take a look at the contents of the repo directory:

repo
├── manifests // manifest repository
│ ├── .git -> ../manifests.git/
│ ├── default.xml // The real warehouse manifest file
│ ├── GLOBAL-PREUPLOAD.cfg
│ └── OWNERS
├── manifests.git // Manifest warehouse git configuration information
│ ├── branches
│ ├── config
│ ├── description
│ ├── FETCH_HEAD
│ ├── HEAD
│ ├── hooks
│ ├── index
│ ├── info
│ ├── logs
│ ├── objects
│ ├── packed-refs
│ └── refs
├── manifest.xml // manifest file, including default.xml file
└── repo // repo tool script
    ├── color.py
    ├── command.py
    ├── git_trace2_event_log.py
    ├── hooks
    ├── hooks.py
    ├── repo
    ├── .........
    └── wrapper.py

repo init does the following things:

  • Download the repo tools repository, which contains python scripts. In the .repo/repo directory.

  • Download the manifests repository, under the directory .repo/manifests, the soft link contains the .repo/manifests.git directory.

  • Downloaded a repository manifest file manifests.xml which contains default.xml.

The content of the repo batch management warehouse is in default.xml. Let's take a look at the file contents:

<?xml version="1.0" encoding="UTF-8"?><manifest>
 <remote name="aosp"
          fetch=".."
          review="https://android-review.googlesource.com/" />
 <default revision="master"
          remote="aosp"
          sync-j="4" />
 <manifest-server url="http://android-smartsync.corp.google.com/android.googlesource.com/manifestserver" />
 <superproject name="platform/superproject" remote="aosp"/>
 <contactinfo bugurl="go/repo-bug" />
 <project path="build/make" name="platform/build" groups="pdk" >
   <linkfile src="CleanSpec.mk" dest="build/CleanSpec.mk" />
   <linkfile src="buildspec.mk.default" dest="build/buildspec.mk.default" />
   <linkfile src="core" dest="build/core" />
   <linkfile src="envsetup.sh" dest="build/envsetup.sh" />
   <linkfile src="target" dest="build/target" />
   <linkfile src="tools" dest="build/tools" />
 </project>
 <project path="build/bazel" name="platform/build/bazel" groups="pdk" >
   <linkfile src="bazel.WORKSPACE" dest="WORKSPACE" />
   <linkfile src="bazel.sh" dest="tools/bazel" />
   <linkfile src="bazel.BUILD" dest="BUILD" />
 </project>
 <project path="build/bazel_common_rules" name="platform/build/bazel_common_rules" groups="pdk" />
 <project path="build/blueprint" name="platform/build/blueprint" groups="pdk,tradefed" />
 <project path="build/pesto" name="platform/build/pesto" groups="pdk" />
 <project path="build/soong" name="platform/build/soong" groups="pdk,tradefed" >
   <linkfile src="root.bp" dest="Android.bp" />
   <linkfile src="bootstrap.bash" dest="bootstrap.bash" />
 </project>
 <project path="art" name="platform/art" groups="pdk" />
 <project path="bionic" name="platform/bionic" groups="pdk" />
 <project path="bootable/recovery" name="platform/bootable/recovery" groups="pdk" />
 <project path="bootable/libbootloader" name="platform/bootable/libbootloader" groups="vts,pdk" />
 <project path="device/common" name="device/common" groups="pdk-cw-fs,pdk" />
 <project path="device/generic/arm64" name="device/generic/arm64" groups="pdk" />
 <project path="device/generic/armv7-a-neon" name="device/generic/armv7-a-neon" groups="pdk" />
 <project path="device/generic/art" name="device/generic/art" groups="pdk" />
 ......
 <repo-hooks in-project="platform/tools/repohooks" enabled-list="pre-upload" />
 <!--
   Merge marker to make it easier for git to merge AOSP-only manifest
   changes to a custom manifest. Please keep this marker here and
   refrain from making changes on or below it.
 --></manifest>  

fold

The syntax of this XML file is not part of this document. Just a few key points:

remote: remote server repo warehouse name
project: git warehouse path of each project
: client git warehouse path
name: server git warehouse name
revision: branch name or commit id
linkfile: soft link file, src and dest represent source and destination .
copyfile: file copy, src and dest indicate source and destination.
groups: grouping.

Summarize the way repo manages warehouses in batches:

  • The repo init command downloads the tool script repository and manifest repository.

  • The repo sync command downloads the remote repository binaries according to the configuration in the manifest file.

  • repo start checks out the branch.

Therefore, the management core of the repo warehouse is manifest.git, and the configuration file that plays a key role is the warehouse list file xxx.xml. There are as many project warehouses as there are corresponding xml list files.

1.2 What do I need to do to build a repo service?

Then comes the key question: what do we need to do to build a repo server?

in short:

  • Deploy the common tool repository git-repo.git.

  • Deploy your own manifest repository manifests.git.

  • Write the manifest file manifests.xml

  • Create project sub-repositories and upload source code in batches.

2 Deployment tool repository git-repo.git

In fact, if your network link can be linked to the external network, the tool repository git-repo.git does not need to be deployed.

Just specify the URL through the REPO_URL environment variable to automatically pull the git-repo.git repository.

export REPO_URL=https://mirrors.tuna.tsinghua.edu.cn/git/git-repo/

Commonly used URLs are:

## google address, need ladder.
https://gerrit.googlesource.com/git-repo
## Tsinghua University Open Source Software Mirror Station
https://mirrors.tuna.tsinghua.edu.cn/git/git-repo/
## Others can search online by yourself...

But in reality, many companies' external networks are restricted and cannot be accessed. At this time, you need to consider building your own tool repository git-repo.git.

The construction of this warehouse is very simple, it is an ordinary git warehouse, but the problem involving GPG signature is more complicated. This process is not part of this document, and documentation on the Internet can be searched.

3 Deployment manifest repository manifests.git

The way to deploy the manifest repository is no different from a normal git repository. The key questions are:

  • How does the inventory warehouse manage multiple project warehouses?

  • How to write manifest file manifests.xml?

3.1 How to design inventory repository and branch?

3.1.1 How many warehouses are needed?

From the server's point of view, your server will definitely manage multiple project repositories (such as qcom8953, qcom8916, mtk6219, mtk2226, rk3399, rk3358, etc.), then the way we manage these repositories may be:

  • The entire server only builds one manifests.git warehouse, and different xml files under each warehouse manage different project warehouses. For example, qcom8953.xml, rk3399.xml, etc.

  • Create a corresponding manifest repository for each project repository, such as manifests-qcom8953.git, manifests-rk3399.git, etc., where the manifest file xml of each project repository is placed.

Either way is fine, depending on your code management strategy. I generally use one manifests.git repository on the entire server, and manage all project repositories through different xml files. The directory structure of the server-side project repository is designed as follows:
aosp
├── manifests.git // manifest repository
│ ├── qcom8953.xml
│ ├── rk3399.xml
│ ├── rk3358.xml
│ └── mtk6226.xml
├ ── qcom/qcom8953 // project warehouse
├── rockchip/rk3399 // project warehouse
├── rockchip/rk3358 // project warehouse
└── mtk/mtk6226 // project warehouse

3.1.2 How many branches are needed?

The manifest repository itself is an ordinary git repository, and it also has branches. The same configuration file, such as qcom8953.xml, can also be split into different branches to manage different project warehouse versions.

The branch of the manifest repository is not directly related to the branch of the project repository. However, in practice, we generally create a branch corresponding to the project warehouse in the inventory warehouse to facilitate the management of the mapping. For example, the qcom8953 project warehouse has four branches master, develop, test and release, then our manifests.git can also create four branches, corresponding to the project warehouse.

I create two branches here: master and release. We will follow these two branches later.

Let's first look at the common parameters of the repo init command:

repo init --helpUsage: repo init [options] [-u] urlOptions:
 Manifest options:
   -u URL, --manifest-url=URL
                       manifest repository location
   -b REVISION, --manifest-branch=REVISION
                       manifest branch or revision (use HEAD for default)
   -m NAME.xml, --manifest-name=NAME.xml
                       initial manifest file
 repo Version options:
   --repo-url=URL repo repository location ($REPO_URL)
   --repo-rev=REV repo branch or revision ($REPO_REV)
   --no-repo-verify do not verify repo source code
 Other options:
   --config-name Always prompt for name/e-mail

In general, the default value for -b is master, and the default value for -m is manifests.xml. If we want to use a different default manifest file, we need to use it together when writing the server manifest file and repo init parameters.

I use all default values here. In the manlifests.xml of the aosp source code, it is reflected in these items:

  <remote name="aosp"
          fetch=".."
          review="https://android-review.googlesource.com/" />
 <default revision="master" ## remote default branch
          remote="aosp" ## The remote default warehouse is name="aosp" in the remote tag
          sync-j="4" />

3.2 How to write a manifest file?

To write the manifest file, it is definitely not possible to write it all by hand, and you need to get an original manifests.xml file. This document is best if you get it from your source code repo.

Here I take qcom's AOSP source code as an example, and modify my own warehouse qcom8953.xml.

<?xml version="1.0" encoding="UTF-8"?><manifest>
   <! --
       remote: Information about the server's remote repository.
       name: The name of the remote manifest warehouse, generally defaults to origin
       fetch: repository url. If it is set to "." (note that it is a dot), it means that it is configured in the repo init -u url.
       review: Review system URL. If you submit code, you will enter this review system.
   -->
   <remote name="origin" fetch="." review="gerrit.your_domain.com" />
       <!--
         default: The default property for this repo repository.
         revision: The default project code repository branch, which can be modified to point to a different branch.
         remote: The name of the remote repository. This corresponds to the <remote> tag, and of course it can also point to remotes in other warehouses.
       -->
       <default revision="master" remote="origin"/>
       <!--
           project: each repository is a project
           path: The path name after the client synchronizes to the local.
           name: The name of the server-side git repository, note the path.
       -->
       <project path="build" name="build" />
       <project path="device" name="device" />
       <project path="frameworks/base" name="frameworks/base" />
       <!--        
           important:
           The build directory and the three subdirectories under it: blueprint, kati, soong are nested git warehouse structures!
           Such nested git repositories. There is a very important control file: .gitignore.
           There must be a .gitignore file in the parent directory, which contains nested child warehouse paths.
       -->
       <project path="build" name="build">
           <!-- copyfile: Copy build/core/root.mk as Makefile. with relative paths -->
           <copyfile src="core/root.mk" dst="Makefile" />
       </project>
       <project path="build/blueprint" name="build/blueprint" />
       <project path="build/kati" name="build/kati" />
       <project path="build/soong" name="build/soong">
           <!-- linkfile: create a soft link Android.bp to build/soong/root.bp -->
           <linkfile src="root.bp" dest="Android.bp" />
           <linkfile src="bootstrap.bash" dest="bootstrap.bash" />
        </project>
        <!-- More repository info... --></manifest>

3.3 Create manifest repository manifests.git

There are two ways to create a manifest repository:

  • Web page creation for the gerrit system. This method is suitable for creating a small number of warehouses, which is simple and fast.

  • gerrit command to create. This method is suitable for creating a large number of warehouses in batches, especially AOSP warehouses.

The first way is relatively simple, you can operate on the WEB page, there is nothing special.

Focus on the second command line method.

ssh -p 29418 <USERNAME>@<GERRIT_IP_ADDRESS> gerrit create-project aosp/manifests.git

After creation, you can also see the manifests.git repository on the gerrit page. Administrators can create two branches on the page: master, release.

The client operates as normal git operations:

## clone code
git clone .../manifests.git
## Submit manifest file qcom8953.xml
git add qcom8953.xml
git commit -m "add qcom8953.xml"
git push

In this way, we have created a manifests.git repository and uploaded a manifest file qcom8953.xml.

The next step is how to create a project repository.

4 Create project sub-warehouses in batches

The source code of the initial project repository, preferably from the complete repo repository with .repo and .git provided by the platform manufacturer, and then we will see how to port this repository to our own internal repository.

General operation steps:

  • The gerrit command creates empty server-side warehouses in batches.

  • Preprocess the local repo repository.

  • Bulk upload source code.

4.1 Batch creation of server-side empty warehouses

Batch creation of sub-repositories can only be done through the gerrit command line.

ssh -p 29418 <USERNAME>@<GERRIT_IP_ADDRESS> gerrit create-project aosp/qcom8953/xxx.git

Here we need to know all the warehouse names of the qcom8953 warehouse, and then create them in batches through scripts. There are two ways:

  • Extract the repository name from the xml script.

  • Get it directly from the project.list file in .repo/manifets.git of the original source code repository.
    Either way, you need to manually check that the final warehouse number and name are consistent.

We ultimately need a repository list file: project.list. The contents of the file are as follows:

qcom/qcom8953/art
qcom/qcom8953/bionic
qcom/qcom8953/bootable/bootloader/edk2
qcom/qcom8953/bootable/recovery
qcom/qcom8953/build/blueprint
qcom/qcom8953/build/kati
qcom/qcom8953/build
qcom/qcom8953/build/soong
qcom/qcom8953/cts
qcom/qcom8953/dalvik
qcom/qcom8953/developers/build
qcom/qcom8953/developers/demos/JustForUs
qcom/qcom8953/developers/samples/android
qcom/qcom8953/development
qcom/qcom8953/device/common
qcom/qcom8953/device/generic/arm64
qcom/qcom8953/device/generic/armv7-a-neon
......

Next we can use the following commands to batch create empty server-side repositories:

# Parameter Description
# --empty-commit: mandatory parameter, must be added;
# -p YOUR-projects: Your gerrit permission repository. Determined according to your gerrit permission name.
# Don't add other parameters. After superfluous, there will be many more problems.
# Note that the path prefix (qcom/qcom8953) is required before the name of the remote warehouse to prevent confusion in the server-side warehouse path.

cat project.list | xargs -I {} ssh -p 29418 [email protected] gerrit create-project "{}" --empty-commit -p YOUR-projects

After the creation, I saw hundreds of git repositories on the gerrit page.

4.2 Preprocessing local repo code

Before uploading the local repo repository, several preprocessing actions need to be done:

  • Delete the .repo directory in the local project.

  • Check the local warehouse .repo/manifests.git/xxxx.xml manifest file, and delete the source of copyfile and linkfile in the local code according to the content of the file.

  • Modify the git remote url path of each local repository. Here you need to write your own script to do it in batches.

git checkout master
git push origin master:master
git push --tags

4.3 Upload branches and code

After the warehouse is preprocessed, you need to upload the specified branches and codes of each warehouse to the server warehouse one by one.
Here is the general operation, you also need to write your own script for upload synchronization:

git checkout mastergit push origin master:mastergit push --tags

Of course, after uploading, you'd better verify in a third-party test if there is a problem with some .gitignore or some nested repositories that need to be repaired manually.

5 Brief description of the use of repo warehouse

After the repo warehouse is built, it is not troublesome for the client to use, and the code submission can be completed by using the commonly used repo commands and git commands.

# Initialize repository
repo init -u <remote manifests url> -n <manifests branch> -m <manifest NAME.xml>
# checkout code
repo sync -j16
# create a local branch
repo start <BRANCH> --all
# Batch execute git commands
repo forall -c `GIT COMMAND`

Code commit merge from a single repository:

git add xxx
git commit -m "xxxx"
git push HEAD:refs/for/BRANCH

A small error of no change id may be reported during the initial submission. Baidu search, it is easy to fix.

Original address: https://www.cnblogs.com/aosp/p/16445394.html


Tags

Technical otaku

Sought technology together

Related Topic

0 Comments

Leave a Reply

+