Quickstart

This quickstart describes the initial setup required to run an instance of Crossfeed on your local computer.

Initial Setup

Mac Users - before starting the initial setup, ensure you have already completed the following: development environment for mac-based computers.
Install Node.js 18 and Docker Compose.
Copy root dev.env.example file to a .env file.
```
cp dev.env.example .env
```
Build the crossfeed-worker Docker image:
```
cd backend && npm run build-worker
```
Start the entire environment from the root directory:
```
npm start
```
Generate the initial DB schema and populate it with sample data:
```
cd backend
# Generate schema
npm run syncdb
# Populate sample data
npm run syncdb -- -d populate
```
If you ever need to drop and recreate the database, you can run npm run syncdb -- -d dangerouslyforce.
Navigate to http://localhost in a browser. The first time please navigate to http://localhost/signup to create account. Local accounts can be set to Global Admin to aide in development.
Hot reloading for source files is enabled, but after changes to non-source code files stopping and starting Docker Compose is required. The following are examples of changes that will require restarting the environment:
- Frontend or backend dependency changes
- Backend changes to serverless.yml or env.yml
- Environment variables in root .env
Install Prettier in your dev environment to format code on save.

Running tests

To run tests, first make sure you have already started Crossfeed with npm start (or, at bare minimum, that the database container is running). Then run:

cd backend
npm test

If snapshot tests fail, update snapshots by running npm test -- -u.

To run tests for the subset of worker code that is written in Python, you need to run:

pip install -r worker/requirements.txt
pytest

To view a code coverage report (a minimum code coverage threshold is checked in CI), run npm test -- --collectCoverage. You can then view a HTML coverage report in the coverage/lcov-report directory.

Monitoring Docker containers

To see which Docker containers are running, you can run:

docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
2a155c5bb9ce        crossfeed_backend                                     "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       0.0.0.0:3000->3000/tcp                           crossfeed_backend_1
0177dff83a80        docker.elastic.co/elasticsearch/elasticsearch:7.9.0   "/tini -- /usr/local…"   13 minutes ago      Up 13 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   crossfeed_es_1
c0f3dee36d5e        crossfeed_frontend                                    "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       0.0.0.0:80->3000/tcp                             crossfeed_frontend_1
f3491e1b547e        matomo:3.14.1                                         "/entrypoint.sh apac…"   13 minutes ago      Up 13 minutes       80/tcp                                           crossfeed_matomo_1
c3ed457a71d2        postgres:latest                                       "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       0.0.0.0:5432->5432/tcp                           crossfeed_db_1
98c14a4f8886        mariadb:10.5                                          "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       3306/tcp                                         crossfeed_matomodb_1
9f70dbdbe867        crossfeed_docs                                        "docker-entrypoint.s…"   13 minutes ago      Up 13 minutes       0.0.0.0:4000->4000/tcp                           crossfeed_docs_1
746956c514ed        bitnami/minio:2020.9.26                               "/opt/bitnami/script…"   13 minutes ago      Up 13 minutes       0.0.0.0:9000->9000/tcp                           crossfeed_minio_1

You can then check the logs of a particular container by specifying a container's name with the docker logs command. For example:

docker logs crossfeed_backend_1 --follow

Further information

To see more information about the design and development of each component of Crossfeed, see the following links:

Frontend for the React frontend.
REST API for the REST API.
Database for the database models stored in Postgres.
Worker for the worker system and adding new scans and data sources.
Search for the search infrastructure and setup with Elasticsearch.
Analytics for the analytics setup with Matomo.

Documentation

The documentation files are stored in the docs directory and served from a Gatsby site. To work on this, you should run npm start from before. You can then open up http://localhost:4000 in your browser to view the docs.

The docs are based on the federalist-uswds-gatsby theme. See that repository for more information on additional theme customizations that can be done.

Common Issues

Node Error issue occurs due to "npm install"

    npm ERR! code EBADENGINE
	npm ERR! engine Unsupported engine
	npm ERR! engine Not compatible with your version of node/npm: crossfeed-backend@1.0.0
	npm ERR! notsup Not compatible with your version of node/npm: crossfeed-backend@1.0.0
	npm ERR! notsup Required: {"node":">=16.0.0 <17.0.0"}
	npm ERR! notsup Actual:   {"npm":"9.5.0","node":"v18.14.2"}

	npm ERR! A complete log of this run can be found in:
    npm ERR!     /Users/combsc/.npm/_logs/2023-03-10T20_01_15_851Z-debug-0.log

In this case install nvm for nodes 16.0.0 to 17.0.0. for example nvm install 16.19.0 then check it by node -- version and npm -- version

Sometimes you may get an error in package-lock.json. This error is due to the package downloading the docker build. Remove the package-lock.json file and reinstall it using npm install.

   rm package-lock.json
   npm install

If successful then continue to step 3.

Permission Issue / Permissions not permitted / Operation not permitted / Module build Failed

   Failed to compile.
	crossfeed-frontend-1  |
	crossfeed-frontend-1  | Error: EPERM: operation not permitted, open '/app/src/index.tsx'
	crossfeed-frontend-1  | ERROR in ./src/index.tsx
	crossfeed-frontend-1  | Module build failed (from ./node_modules/source-map-loader/dist/cjs.js):
	crossfeed-frontend-1  | Error: EPERM: operation not permitted, open '/app/src/index.tsx'
	crossfeed-frontend-1  |
	crossfeed-frontend-1  | ERROR in [eslint] EPERM: operation not permitted, open '/app/src/index.tsx'
	crossfeed-frontend-1  |
	crossfeed-frontend-1  | webpack compiled with 2 errors
    crossfeed-frontend-1  | No issues found.

If you receive the above error check the following:

   system settings > Privacy & Security > Files and Folder > "Find Docker" > click on Docker > Under Documents Folder ensure its on by swipping right and showing blue icon.

If the above is correct try looking into the development environment for mac-based computers. Then Review the account permissions as necessary.