GitLab

Availability:

Airbyte Cloud Airbyte OSS PyAirbyte

Support Level:

Airbyte Connector

Connector Version:

4.3.3(Last updated 3 months ago)

CDK Version:

4.5.0

Sync Success Rate:

Usage Rate:

This page contains the setup guide and reference information for the Gitlab Source connector.

• Gitlab instance or an account at Gitlab

For Airbyte Cloud:

• Personal Access Token (see personal access token)
• OAuth

For Airbyte Open Source:

• Personal Access Token (see personal access token)

Create a GitLab Account or set up a local instance of GitLab.

Airbyte Open Source additional setup steps

Log into GitLab and then generate a personal access token. Your token should have the read_api scope, that Grants read access to the API, including all groups and projects, the container registry, and the package registry.

For Airbyte Cloud:

1. Log into your Airbyte Cloud account.
2. In the left navigation bar, click Sources. In the top-right corner, click + new source.
3. On the source setup page, select GitLab from the Source type dropdown and enter a name for this connector.
4. Click Authenticate your GitLab account by selecting Oauth or Personal Access Token for Authentication.
5. Log in and Authorize to the GitLab account.
6. API URL (Optional) - The URL to access your self-hosted GitLab instance or gitlab.com (default).
7. Start date (Optional) - The date from which you'd like to replicate data for streams.
8. Groups (Optional) - List of GitLab group IDs, e.g. airbytehq for single group, airbytehq another-repo for multiple groups.
9. Projects (Optional) - List of GitLab projects to pull data for, e.g. airbytehq/airbyte.
10. Click Set up source.

Note: You can specify either Group IDs or Project IDs in the source configuration. If both fields are blank, the connector will retrieve a list of all the groups that are accessible to the configured token and ingest as normal.

For Airbyte Open Source:

1. Authenticate with Personal Access Token.

The Gitlab Source connector supports the following sync modes:

• Full Refresh - Overwrite
• Full Refresh - Append
• Incremental - Append
• Incremental - Append + Deduped

This connector outputs the following streams:

• Branches
• Commits (Incremental)
• Issues (Incremental)
• Group Issue Boards
• Pipelines (Incremental)
• Jobs
• Projects
• Project Milestones
• Project Merge Requests (Incremental)
• Users
• Groups
• Group Milestones
• Group and Project members
• Tags
• Releases
• Deployments
• Group Labels
• Project Labels
• Epics (only available for GitLab Ultimate and GitLab.com Gold accounts. Stream Epics uses iid field as primary key for more convenient search and matching with UI. Iid is the internal ID of the epic, number of Epic on UI.)
• Epic Issues (only available for GitLab Ultimate and GitLab.com Gold accounts)

GitLab source works with GitLab API v4. It can also work with self-hosted GitLab API v4.

Gitlab has the rate limits, but the Gitlab connector should not run into Gitlab API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.

Config fields reference

Field

Type

Property name

object

credentials

string

start_date

string

api_url

string

groups

array<string>

groups_list

string

projects

array<string>

projects_list

Expand to review

Version	Date	Pull Request	Subject
4.3.3	2024-08-17	44207	Update dependencies
4.3.2	2024-08-12	43856	Update dependencies
4.3.1	2024-08-03	43058	Update dependencies
4.3.0	2024-07-31	42920	Migrate to CDK v4.1.0
4.2.2	2024-07-27	42601	Update dependencies
4.2.1	2024-07-20	42295	Update dependencies
4.2.0	2024-07-17	42085	Migrate to CDK v2.4.0
4.1.0	2024-07-17	42021	Migrate to CDK v1.8.0
4.0.8	2024-07-13	41835	Update dependencies
4.0.7	2024-07-10	41470	Update dependencies
4.0.6	2024-07-09	41100	Update dependencies
4.0.5	2024-07-06	40894	Update dependencies
4.0.4	2024-06-25	40417	Update dependencies
4.0.3	2024-06-22	40102	Update dependencies
4.0.2	2024-04-24	36637	Schema descriptions and CDK 0.80.0
4.0.1	2024-04-23	37505	Set error code 500 as retryable
4.0.0	2024-03-25	35989	Migrate to low-code
3.0.0	2024-01-25	34548	Fix merge_request_commits stream to return commits for each merge request
2.1.2	2024-02-12	35167	Manage dependencies with Poetry.
2.1.1	2024-01-12	34203	prepare for airbyte-lib
2.1.0	2023-12-20	33676	Add fields to Commits (extended_trailers), Groups (emails_enabled, service_access_tokens_expiration_enforced) and Projects (code_suggestions, model_registry_access_level) streams
2.0.0	2023-10-23	31700	Add correct date-time format for Deployments, Projects and Groups Members streams
1.8.4	2023-10-19	31599	Base image migration: remove Dockerfile and use the python-connector-base image
1.8.3	2023-10-18	31547	Add validation for invalid groups_list and/or projects_list
1.8.2	2023-10-17	31492	Expand list of possible error status codes when handling expired access_token
1.8.1	2023-10-12	31375	Mark start_date as optional, migrate groups and projects to array
1.8.0	2023-10-12	31339	Add undeclared fields to streams schemas, validate date/date-time format in stream schemas
1.7.1	2023-10-10	31210	Added expired access_token handling, while checking the connection
1.7.0	2023-08-08	27869	Add Deployments stream
1.6.0	2023-06-30	27869	Add shared_runners_setting field to groups
1.5.1	2023-06-24	27679	Fix formatting
1.5.0	2023-06-15	27392	Make API URL an optional parameter in spec.
1.4.2	2023-06-15	27346	Partially revert changes made in version 1.0.4, disallow http calls in cloud.
1.4.1	2023-06-13	27351	Fix OAuth token expiry date.
1.4.0	2023-06-12	27234	Skip stream slices on 403/404 errors, do not fail syncs.
1.3.1	2023-06-08	27147	Improve connectivity check for connections with no projects/groups
1.3.0	2023-06-08	27150	Update stream schemas
1.2.1	2023-06-02	26947	New field name added to Pipelines and PipelinesExtended stream schema
1.2.0	2023-05-17	22293	Preserve data in records with flattened keys
1.1.1	2023-05-23	26422	Fix error 404 Repository Not Found when syncing project with Repository feature disabled
1.1.0	2023-05-10	25948	Introduce two new fields in the Projects stream schema
1.0.4	2023-04-20	21373	Accept api_url with or without scheme
1.0.3	2023-02-14	22992	Specified date formatting in specification
1.0.2	2023-01-27	22001	Set AvailabilityStrategy for streams explicitly to None
1.0.1	2023-01-23	21713	Fix missing data issue
1.0.0	2022-12-05	7506	Add OAuth2.0 authentication option
0.1.12	2022-12-15	20542	Revert HttpAvailability changes, run on cdk 0.15.0
0.1.11	2022-12-14	20479	Use HttpAvailabilityStrategy + add unit tests
0.1.10	2022-12-12	20384	Fetch groups along with their subgroups
0.1.9	2022-12-11	20348	Fix 403 error when syncing EpicIssues stream
0.1.8	2022-12-02	20023	Fix duplicated records issue for Projects stream
0.1.7	2022-12-01	19986	Fix GroupMilestones stream schema
0.1.6	2022-06-23	13252	Add GroupIssueBoards stream
0.1.5	2022-05-02	11907	Fix null projects param and container_expiration_policy
0.1.4	2022-03-23	11140	Ingest All Accessible Groups if not Specified in Config
0.1.3	2021-12-21	8991	Update connector fields title/description
0.1.2	2021-10-18	7108	Allow all domains to be used as api_url
0.1.1	2021-10-12	6932	Fix pattern field in spec file, remove unused fields from config files, use cache from CDK
0.1.0	2021-07-06	4174	Initial Release