66 lines
1.4 KiB
Markdown
66 lines
1.4 KiB
Markdown
|
# Job Scraper
|
||
|
|
|
||
|
|
Job board monitoring for privacy-focused and open-source companies.
|
||
|
|
|
||
|
|
## Quick Reference
|
||
|
|
|
||
|
|
| Item | Value |
|
||
|
|
|------|-------|
|
||
|
|
| URL | https://jobs.novanexus.ca |
|
||
|
|
| Port | 8085 |
|
||
|
|
| Containers | job-scraper-scheduled, job-dashboard |
|
||
|
|
| Data | ./data/jobs.db, ./data/dashboard.html |
|
||
|
|
|
||
|
|
## Development Workflow
|
||
|
|
|
||
|
|
After making code changes, rebuild and deploy:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd ~/job-scraper && sudo docker compose build && sudo docker compose up -d
|
||
|
|
```
|
||
|
|
|
||
|
|
## Common Commands
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# View status
|
||
|
|
sudo docker compose ps
|
||
|
|
|
||
|
|
# View logs
|
||
|
|
sudo docker compose logs -f scraper-scheduled
|
||
|
|
|
||
|
|
# Run scraper manually (one-time)
|
||
|
|
sudo docker compose run --rm scraper
|
||
|
|
|
||
|
|
# Restart services
|
||
|
|
sudo docker compose restart
|
||
|
|
|
||
|
|
# Rebuild without cache (if having issues)
|
||
|
|
sudo docker compose build --no-cache && sudo docker compose up -d
|
||
|
|
```
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
- `config.yaml` - Companies to monitor, filters, notifications
|
||
|
|
- After config changes: `sudo docker compose restart scraper-scheduled`
|
||
|
|
|
||
|
|
## Email Notifications
|
||
|
|
|
||
|
|
Uses msmtp with system config (`~/.msmtprc`). The container mounts this file.
|
||
|
|
|
||
|
|
To test email manually:
|
||
|
|
```bash
|
||
|
|
sudo docker compose run --rm scraper
|
||
|
|
```
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
- `main.py` - CLI entry point, scheduling
|
||
|
|
- `db.py` - SQLite database operations
|
||
|
|
- `notify.py` - Console, msmtp, email, Slack notifications
|
||
|
|
- `dashboard.py` - HTML dashboard generator
|
||
|
|
- `scrapers/` - Platform-specific scrapers (greenhouse, lever, ashby)
|
||
|
|
|
||
|
|
## Documentation
|
||
|
|
|
||
|
|
Full docs: ~/maple-docs/docs/04-services/job-scraper.md
|