Get Me There - Bus Intelligent Transportation System (BITS)

An innovative solution that uses IOT, Microsoft Azure and Data Analytics to save passengers' time.

AdvancedFull instructions providedOver 1 day2,856

Things used in this project

Hardware components

Android device
Android device
used to test the Android App
×1
Raspberry Pi 3 Model B
Raspberry Pi 3 Model B
×1
Breadboard (generic)
Breadboard (generic)
×1
Jumper wires (generic)
Jumper wires (generic)
×14
Push-Button
×3
LED (generic)
LED (generic)
×3
Resistor 221 ohm
Resistor 221 ohm
×3

Software apps and online services

Android Studio
Visual Studio 2015
Microsoft Visual Studio 2015
Windows 10 IoT Core
Microsoft Windows 10 IoT Core
Microsoft Windows IoT Core Project Templates on VS
Microsoft Windows IOT Core Dashboard
Microsoft Azure
Microsoft Azure
Microsoft Azure Storage Explorer
Microsoft Azure Storage Explorer (Preview) is a standalone app from Microsoft that allows you to easily work with Azure Storage data on Windows, macOS and Linux.
Microsoft Power BI
Power BI is a suite of business analytics tools to analyze data and share insights. Monitor your business and get answers quickly with rich dashboards available on every device.
Microsoft Device Explorer
Microsoft Azure IOT Hub
Microsoft Azure App Service plan
Microsoft Azure Machine Learning Studio
Microsoft Azure Machine Learning Web Service Plan
Microsoft Azure Storage
Microsoft Azure Virtual machine
Microsoft Azure SQL server
Microsoft Azure Stream Analytics job
Google Maps Distance Matrix and Directions API
Fritzing
Apache Spark
Apache Hive
Apache Impala
Microsoft power bi
Cloudera ODBC

Story

Read more

Schematics

IOT Device - Prototype circuit

The prototype rasperry Pi circuit used to simulate data for IN/OUT passengers and bus's door open/close states

Code

busDataSetEdit.py

Python
A script to transform the DataSet by added four columns to each output files, the new columns will match the following:
a. Pass_IN: Random number of passengers getting in the bus at each station
b. Pass_Out: random number of passengers getting at from the bus at each station
c. Total_Pass: total number of passengers onboard of the bus
d. timestamp: change the prod time which is Unix time to Data Time
• Assumption: bus maximum on board number of passengers is 60
#! /usr/bin/env python
# Auther: Mohamed Moussa

import csv
import random
import datetime

# define the input file nd output file
businfoInput = open('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_2013_01_07.csv', 'r')
businfoOutput = open('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01_07.csv', 'w')
businfoReader = csv.reader(businfoInput)
businfoWriter = csv.writer(businfoOutput, lineterminator='\n')

#define maximum number of passengers on board
maxTotal = 60
journyId=0
businfoReader.next()

# loop in the file to add four new columns: Passengers IN, Passengers Out, Number of onboard passengers, Daytime
for row in businfoReader:
	# first station & and applied on all new journyes
	if (row[3] != journyId):
		Utime=int(row[0])/1000000
		# convert Unix time into date time
		Dtime=datetime.datetime.utcfromtimestamp(
       			int(Utime)
    			).strftime('%H:%M:%S')
		journyId = row[3]
		#partitionKey & RowKey columns required n order to upload the file to Micosfot Azure table
		PartitionKey=row[3]
		RowKey=row[7]
		firstIn = random.randint(0, maxTotal)
		firstOut = 0
		onBoard= firstIn
		# write output to the output file
       	 	businfoWriter.writerow([PartitionKey, RowKey]+row+[firstIn, firstOut, onBoard, Dtime])
	else:
		# used for all sttions with the same journy
               	passOut = random.randint(0,onBoard)
       	 	passIn = random.randint(0, maxTotal-(onBoard-passOut))
        	onBoard = (onBoard + passIn) - passOut
		#partitionKey & RowKey columns required n order to upload the file to Micosfot Azure table
                PartitionKey=row[3]
                RowKey=row[7]
		Utime=int(row[0])/1000000
       	 	Dtime=datetime.datetime.utcfromtimestamp(
       	         	int(Utime)
       	         	).strftime('%H:%M:%S')
       	 	businfoWriter.writerow([PartitionKey, RowKey]+row+[passIn, passOut, onBoard, Dtime])

#close the opened files
businfoInput.close()
businfoOutput.close()

busjournyStation.py

Python
to extract the longest journeys based on number of stations
#Script to extract list of longest & shortest Journyes by number of stations
#Auther: Mohamed Moussa

from pyspark import SparkContext, SparkConf
import csv
import glob
import os

conf = SparkConf().setAppName('BusinfoJournyByStation')
sc = SparkContext(conf=conf)

for filename in glob.iglob('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01*'): #reads files from local FS
	fn=filename.split("/")
        fullname=fn[6]
	# to read files from local file system instead of Hadoop
	localFileName="file://"+filename

	trackFile= sc.textFile(localFileName)


	#Function to parse the file into tuples of journyID & Bus Station
	def makeTrackFile(line):
	        l= line.split(",")
	      	#lDate=l[4]
	        lJournyId=l[5]
	        lStation=l[9]
	        return (lJournyId,lStation)

	#filter for values that has na!=None, means to remove all records for the stations that the bus didnt stop at
	fileLines= trackFile.map(lambda line: makeTrackFile(line))
	#select Distinct over the journyID, then reduceByKey to get total number of stations, then add the date then sort the output and remove brackets
	totalByStation = fileLines.distinct().map(lambda x: (x[0],1)).reduceByKey(lambda x,y:(x+y)).map(lambda x: (x[1],x[0])).sortByKey(ascending=False).map(lambda (k, v) : "{0}, {1}".format(k, v))

	#Save output to file - local filesystem
	totalByStation.saveAsTextFile('file:/home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/' + fullname)

buscount.py

Python
to extract list of buses and journey ID
#Script to extract list of buses and Journyes 
# Auther: Mohamed Moussa

from pyspark import SparkContext, SparkConf
import csv
import glob
import os

conf = SparkConf().setAppName('BusCount')
sc = SparkContext(conf=conf)

for filename in glob.iglob('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01_*'): #reads files from local FS
	fn=filename.split("/")
        fullname=fn[6]
	# to read files from local file system instead of Hadoop
	localFileName="file://"+filename

	trackFile= sc.textFile(localFileName)

	#Function to parse the file into tuples of journyID & Bus Station
	def makeTrackFile(line):
	        l= line.split(",")
		lJourneyID=l[5]
	        lbusId=l[8]
	        return (lbusId, lJourneyID)

	fileLines= trackFile.map(lambda line: makeTrackFile(line))
	#select Distinct over the BusID
	totalByBuses = fileLines.distinct().map(lambda (k, v) : "{0}, {1}".format(k, v))

	#Save output to file - local filesystem
	totalByBuses.saveAsTextFile('file:/home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/Buses/' + fullname)
	#os.system("for i in `ls /home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/*.csv/p*`; do cat $i >> /home/cloudera/DublinBuses010113-310113/Output/JournyStationFull.csv ; done")

BITSCode

API App Project Code

Credits

Abdulrahman Elsharqawy

Abdulrahman Elsharqawy

2 projects • 5 followers
A passionate Maker
Mohamed Hassan AbdulRahman

Mohamed Hassan AbdulRahman

4 projects • 21 followers
Maker, aiming to make people’s life easier and more meaningful, using the latest HW & SW technologies, analysis and research.
Ahmed Emam

Ahmed Emam

1 project • 4 followers
PM/Developer/Maker/ITIL Expert
Mohamed Moussa

Mohamed Moussa

1 project • 1 follower
Ahmed Hassan

Ahmed Hassan

1 project • 1 follower
Hacker ^--^
Waleed Abdel Fattah

Waleed Abdel Fattah

1 project • 1 follower
Maker

Comments