Development Workflow
CI/CD Workflows
Deployment to Fennel can be done via Github actions (or any other post-merge)
script. Credentials to Fennel servers are typically injected as environment
variables in this script such that when commit is called, the client ends up
connecting with the chosen Fennel servers.
Example: Github Actions
Say we have a single dataset as follows in a file called datasets.py:
1from fennel.datasets import dataset, field
2from fennel.lib import meta
3from fennel.connectors import source, Webhook
4
5@meta(owner="[email protected]")
6@source(
7 Webhook(name="example").endpoint("ticket_sale"),
8 disorder="14d",
9 cdc="upsert",
10)
11@dataset(index=True)
12class Ticket:
13 ticket_id: str = field(key=True)
14 price: int
15 at: datetimepython
And a single featureset as follows in a file called featuresets.py:
1from fennel import featureset, feature as F
2
3
4@featureset
5class TicketFeatures:
6 ticket_id: str
7 price: int = F(Ticket.price, default=0)python
A simple Python script could be written that imports both the dataset & featureset
modules, reads Fennel URL & Token as environment variables and calls commit.
1import os
2from fennel.client import Client
3
4from ci_cd.datasets import Ticket
5from ci_cd.featuresets import TicketFeatures
6
7if __name__ == "__main__":
8 url = os.getenv("FENNEL_URL")
9 token = os.getenv("FENNEL_TOKEN")
10
11 client = Client(url=url, token=token)
12 client.commit(
13 message="ticket: add initial dataset and featureset",
14 datasets=[Ticket],
15 featuresets=[TicketFeatures],
16 )python
Given such a setup, here is a sample Github actions script to deploy changes to Fennel servers:
1name: commit
2
3on: # deploy only on pushes to the main branch
4 push:
5 branches:
6 - main
7
8jobs:
9 fennel-commit:
10 runs-on: ubuntu-latest
11 name: Commit datasets and features to Fennel cluster.
12 steps:
13 - name: Checkout Repository
14 uses: actions/checkout@v2
15
16 - name: Setup Python
17 uses: actions/setup-python@v2
18 with:
19 python-version: 3.11.0
20
21 # Depending on your setup of Fennel, you may need to setup a VPN
22 # connection here to talk to Fennel from Github action workers.
23 # You may also need to install more pip packages here depending
24 # on what else is needed to import your Python modules.
25
26 # Install fennel and commit against the cluster
27 - name: Commit Datasets and Features
28 env:
29 FENNEL_URL: ${{ secrets.FENNEL_URL }}
30 FENNEL_TOKEN: ${{ secrets.FENNEL_TOKEN }}
31 run: |
32 python3 -m pip install --upgrade pip
33 pip install "fennel-ai>=0.19.0"
34 python3 -m path.to.fennel.code.commitbash
Deployment Variations
- Separate environments for dev, staging, prod etc. - you can create multiple Fennel deployments, one each for dev, staging, prod etc. Each of these will have their own URL & tokens. You can modify the script to just inject the URL/token of an appropriate deployment at the right time.
- Canary based deployment - you can modify the above script to first deploy to a canary deployment of Fennel (by injecting appropriate URL/token), run whatever sanity checks you want to run, and then proceed with deployment to prod after that.
- Post-code review deployments - you can change the trigger of the script above such that it runs only at a stage where all code reviews have been resolved (e.g. changing the step or branch name)
- Disabling prod deployment outside of CI/CD - Fennel supports fined-grained RBAC using which you can control permissions of tokens. For instance, you can specify that only the tokens of a particular role (say super admin) can deploy to Prod and all other tokens can only read feature values. With this, as long as that token is only available in your CI/CD environment, your team members can still see/query prod Fennel but not modify it without having gone through CI/CD first.
Stateful Rollbacks
Unlike Git, Fennel doesn't have a direct operation for reverting deployments. This is the case because dataset/featureset changes create some state - for instance, some data might have been ingested in a dataset already - should it be deleted during roll back or not. In that sense, it is somewhat similar to roll backs of database schema migrations.
Consequently, the best way to roll back is to not blindly revert the commit but rather explicitly mark things as deprecated or deleted before committing again.
To prevent accidental deletions of entities, Fennel requires you to first do a
commit with datasets/featuresets marked as deleted and then in subsequent commit,
you are free to omit the entities from the commit operation. Here is a simple
example showing how to mark dataset (or featureset) as deleted.
1from fennel.datasets import dataset, field
2from fennel.lib import meta
3from fennel.connectors import source, Webhook
4
5@meta(owner="[email protected]", deleted=True)
6@source(
7 Webhook(name="example").endpoint("ticket_sale"),
8 disorder="14d",
9 cdc="upsert",
10)
11@dataset(index=True)
12class Ticket:
13 ticket_id: str = field(key=True)
14 price: int
15 at: datetimepython