Webcam IoT DIY

Have you wondered how the IP or CCTV cameras are built, configured, or developed? I discovered that there are common open source tools that enable the IP camera features. I want to walk you through how to build a security surveillance system at home. This solution can scale to be a enterprise grade deployment but requires sophisticated configuration automaton software such as Ansible for configuring the Linux IoT devices and the edge servers that processes or stores the recorded video files.

Start simple with periodic image capture from the USB webcam

The use case here isn’t to monitor or record continuously, but to use office webcams to record user’s facial expressions. I found it quite interesting to review the pictures taken to understand how hard or how much I worked. If a meeting such as Google Meet is using /dev/video0, the script will fail to take a picture.

The script to capture webcam images depends on /dev/video0 which is usually the webcam’s device path for the 1st USB webcam. I’m using a Airhug webcam. Install ffmpeg first and replace the environment variables for where you want to put the log file and captured images. The log file gets appended with execution datetime with 1>> $LOG_FILE 2>&1. Replace $HOME with the actual value if executing from a cron job.

#!/bin/bash
set -e # exit the script when execution hits any error
set -x # print the executing lines
LOG_FILE=$HOME/Documents/webcam.log
IMAGE_SINK_FOLDER=$HOME/Pictures
WEBCAM_DEVICE=/dev/video0
DATETIME=$(date +"%Y-%m-%d_%H-%M-%S")
IMAGE_SINK_FILE=$IMAGE_SINK_FOLDER/$DATETIME.jpg
echo -e "\n===   $DATETIME   ===" >> $LOG_FILE
date >> $LOG_FILE
ffmpeg -f v4l2 -video_size 1280x720 -i $WEBCAM_DEVICE -frames 1 \
$IMAGE_SINK_FILE 1>> $LOG_FILE 2>&1

Save the script above as webcam.sh. Execute chmod +x webcam.sh to make it executable. Test it to make sure no error. Execute crontab -e to add a line for every half an hour execution: */30 * * * * /home/hil/bin/webcam.sh . I noticed that there were blurry pictures taken from Logitech Webcam C925e. I suspect the webcam failed to auto-focus when the picture was taken.

Script to configure the motion webcam server

Motion is a program that works as the webcam server. Most IP cameras use it. I configured it on a Raspberry Pi 4 with a FUMAX USB webcam. I’m guessing it works the same on x86 or AMD64 computers. Install Motion (if not exist), arecord , then configure /etc/motion/motion.conf. Here’s the diff between the original motion.conf and my changes. I will explain each setting. Line of < was the default, > is my setting. I executed lsusb -s 001:002 -v | egrep ‘Width|Height’ to get the webcam’s resolutions.

11c11
< daemon off
---
> daemon on
79c79
< width 320
---
> width 1280
82c82
< height 240
---
> height 720
86c86
< framerate 2
---
> framerate 30
104c104
< netcam_keepalive off
---
> netcam_keepalive on
131c131
< auto_brightness off
---
> auto_brightness on
259c259
< quality 75
---
> quality 85
413c413
< snapshot_filename %v-%Y%m%d%H%M%S-snapshot
---
> snapshot_filename %v__%Y-%m-%d_%H-%M-%S_snapshot
422c422
< picture_filename %v-%Y%m%d%H%M%S-%q
---
> picture_filename %v__%Y-%m-%d_%H-%M-%S_%q
427c427
< movie_filename %v-%Y%m%d%H%M%S
---
> movie_filename %v__%Y-%m-%d_%H-%M-%S
432c432
< timelapse_filename %Y%m%d-timelapse
---
> timelapse_filename %Y-%m-%d_timelapse
445c445
< stream_port 8081
---
> stream_port 8050
458c458
< stream_maxrate 1
---
> stream_maxrate 30
461c461
< stream_localhost on
---
> stream_localhost off
472c472
< stream_auth_method 0
---
> stream_auth_method 1
> webcontrol_auth_method 1
476c476
< ; stream_authentication username:password
---
> stream_authentication user:password
492c492
< webcontrol_port 8080
---
> webcontrol_port 8051
495c495
< webcontrol_localhost on
---
> webcontrol_localhost off
502c502
< ; webcontrol_authentication username:password
---
> webcontrol_authentication root:password
601c601
< ; on_event_start value
---
> on_event_start arecord -f S16_LE -r 48000 -D hw:2,0  /var/lib/motion/%v__%Y-%m-%d_%H-%M-%S.wav
605c605
< ; on_event_end value
---
> on_event_end ps -ef|grep arecord|grep "hw:2,0"|grep -v grep|awk '{print $2}'|xargs kill -9

daemon: on to start motion process and return immediately vs waiting on the terminal for Ctrl+C. To kill the process, I’ve tried services motion stop or killall motion.
width, height: change the width and height to be a typical 1080p webcam’s w,h.
framerate, stream_maxrate: 30 is fine for pi 4’s processor. Higher number requires more processing power. Each webcam may have a max hardware fps limit.
netcam_keepalive: on to optimize for HTTP1.1 streaming.
auto_brightness: on to avoid dark videos
quality: I set it to +10 higher
snapshot_filename, picture_filename, movie_filename: I’m using %v__ to merge the video and audio files. The later script will depend on this naming convention.
*_port: set the port that does not conflict with some commonly used ports.
stream_localhost, webcontrol_localhost: off means allow connection from other than localhost.
stream_auth_method, webcontrol_auth_method : 1 means basic authentication. The comment in the .conf file has more details
webcontrol_authentication, stream_authentication: username, password separated by :
on_event_start: arecord command to use ALSA to record audio wav files. Execute arecord -l to get the audio input device. arecord has been tested to work on Raspberry Pi 4 model B but failed on Ubuntu 22.04 desktop edition. Error indicated no usable audio device. I guess GDM3 was using it.

**** List of CAPTURE Hardware Devices ****
card 2: U20 [USB PHY 2.0], device 0: USB Audio [USB Audio]
 Subdevices: 1/1
 Subdevice #0: subdevice #0

card 2, device 0 makes -D hw:2,0 as in arecord -f S16_LE -r 48000 -D hw:2,0 /var/lib/motion/%v__%Y-%m-%d_%H-%M-%S.wav. The script below merges the video file with the audio file here which depends on %v__ in formatting.
on_event_end: the command to kill arecord. Somehow kill -9 is required here although normal kill works on the command terminal in manual testing.

I executed motion as root to start the daemon. There may be a more secure method to run it as a normal user but I did not bother. The command should return immediately. You should see files at /var/lib/motion/ generated for the videos.

Configure iWatch to execute the rsync command

Identify a remote archive server where the SSH public key is stored under /home/$REMOTE_USER/.ssh/authorized_keys. Test with ssh remote-user@SERVER_IP assuming the remote server IP is 192.168.1.10 and it has writable path at DEST_DIR . ssh command needs to succeed without entering the password or confirm hostname. Install iwatch :

sudo apt update && \
sudo apt install iwatch && \
sudo su -

Running as root, configure iwatch xml at /etc/iwatch/iwatch.xml to watch the directory motion process dumps video files: /var/lib/motion/*.mkv ;

<?xml version="1.0" ?>
<!DOCTYPE config SYSTEM "/etc/iwatch/iwatch.dtd" >

<config charset="utf-8">
  <guard email="hil@localhost" name="webcam_files"/>
  <watchlist>
    <title>motion</title>
    <contactpoint email="hil@localhost" name="Administrator"/>
    <path type="single" events="close_write" filter="\.mkv$" exec="/usr/local/bin/move-motion-files.sh %f">/var/lib/motion</path>
  </watchlist>
</config>

Create the shell script to call when a video file is finalized for the write event. Assuming the file server is 192.168.1.2 which has user ipcam.

# The 1st argument is the full path to the file generated by the motion webcam

set -e # exit the script when execution hits any error

USER=ipcam && FILE_SERVER=192.168.1.2 && \
DEST_DIR=/mnt/1tb/ftp/ipcam/autodelete/webcams/ryzen5 && \
rsync -av --remove-source-files -e ssh $1 $USER@$FILE_SERVER:$DEST_DIR/

Start the iwatch daemon as root:

sudo iwatch -d -f /etc/iwatch/iwatch.xml && \
ps -A|grep iwatch
# stop the daemon
killall -v iwatch

Test manually copying a .mkv file to /var/lib/motion/ or trigger motion to write a file there. watch ls /var/lib/motion/ command will show the file appear for writing and gone as it got moved to the remote file server. Test the remote file server’s $DEST_DIR has the file.

Alternatively, create a cron job to archive video files to a remote server

Warning: the following script is applicable to Raspberry Pi 4 model B where arecord can generate audio files. Cronjob isn’t the right method because the video file could be open for write but not finalized to cause copy failure.

Identify a remote archive server where the SSH public key is stored under /home/$REMOTE_USER/.ssh/authorized_keys. Test with ssh remote-user@SERVER_IP assuming the remote server IP is 192.168.1.10 and it has writable path at DEST_DIR. Confirm the environment variables are applicable to the webcam server. For example, create VIDEOS_DIR if it does not exist.

#!/bin/bash
# Merge mkv videos generated by the Linux motion https://motion-project.github.io/motion_config.html with audio
# /etc/motion/motion.conf should include the recording commands for the correct format:
# >>> on_event_start arecord -f S16_LE -r 48000 -D hw:1,0  /var/lib/motion/%v__%Y-%m-%d_%H-%M-%S.wav
# >>> on_event_end ps -ef|grep arecord|grep "hw:1,0"|grep -v grep|awk '{print $2}'|xargs kill -9

# Usage: Change environment variables; make sure they exist; upload SSH public key to the user on the file server
# test ssh $USER@$FILE_SERVER without password; verify folder DEST_DIR exists on the file server

set -x # print the executing lines
set -e # exit the script when execution hits any error

WORKING_DIR=/var/lib/motion
VIDEOS_DIR=/root/Videos
DEST_DIR=/mnt/1tb/ftp/michelle/webcams/pi
FILE_SERVER=192.168.1.10
USER=michelle

# only process files that has a sequence before __ in filename
FILES=$(find $WORKING_DIR -name "*__*.mkv" -type f ! -exec fuser -s "{}" 2>/dev/null \; -exec echo {} \;)

for f in $FILES
do
  # get the audio file which could be written 1 second later than the filename 2022-09-10_21-22-29.mkv
  # use the %v__ in filename for event sequence. For example,
  # /var/lib/motion/01__2022-09-10_22-06-21.mkv /var/lib/motion/01__2022-09-10_22-06-21.wav
  # split the filepath by the underscore character '_' before which is the event sequence
  IFS='_' read -r -a SEQ <<< "$f"
  # check file ${SEQ[0]}*.wav exists
  AUDIO_FILE=${SEQ[0]}*.wav
  if ls $AUDIO_FILE 1> /dev/null 2>&1; then
    echo "Found corresponding audio file: $AUDIO_FILE"
  else
    # stop processing the current .mkv file as there is no corresponding .wav file
    continue
  fi

  # replace the file path's working dir substring with videos dir string
  merged_file=${f/$WORKING_DIR/$VIDEOS_DIR}
  # mkv video tool to merge video.mkv with audio.wav
  mkvmerge -o $merged_file -A $f $AUDIO_FILE
  # remove source files after successful video audio merge
  rm $f $AUDIO_FILE
  # archive the files to remote servers
  rsync -av --remove-source-files -e ssh $VIDEOS_DIR/* $USER@$FILE_SERVER:$DEST_DIR/
done

Execute chmod a+x archive-webcam-pix.sh and test it before adding a line in cron job for execution every 10th minute: */10 * * * * /root/archive-webcam-pix.sh >> /root/archive-webcam-pix.log 2>&1

Auto delete files recursively on the remote archive server

The best practice is to put the archive server at a secure location and set DEST_DIR to a mounted path. Review the comments in autodelete.py to

1. Configure the python utility to auto delete files that are 1 week older.
2. Configure bash commands to delete empty directories in a cron job.