Things used in this project

Hardware components:
1434509556 android phone color
Android device
used to test the Android App
×1
Pi 3 02
Raspberry Pi 3 Model B
×1
12002 04
Breadboard (generic)
×1
11026 02
Jumper wires (generic)
×14
Push-Button
×3
09590 01
LED (generic)
×3
Mfr 25fbf52 221r sml
Resistor 221 ohm
×3
Software apps and online services:
Android Studio
Vs2015logo
Microsoft Visual Studio 2015
10
Microsoft Windows 10 IoT Core
Microsoft Windows IoT Core Project Templates on VS
Microsoft Windows IOT Core Dashboard
W9gt7hzo
Microsoft Azure
Microsoft Azure Storage Explorer
Microsoft Azure Storage Explorer (Preview) is a standalone app from Microsoft that allows you to easily work with Azure Storage data on Windows, macOS and Linux.
Microsoft Power BI
Power BI is a suite of business analytics tools to analyze data and share insights. Monitor your business and get answers quickly with rich dashboards available on every device.
Microsoft Device Explorer
Microsoft Azure IOT Hub
Microsoft Azure App Service plan
Microsoft Azure Machine Learning Studio
Microsoft Azure Machine Learning Web Service Plan
Microsoft Azure Storage
Microsoft Azure Virtual machine
Microsoft Azure SQL server
Microsoft Azure Stream Analytics job
Google Maps Distance Matrix and Directions API
Fritzing
Apache Spark
Apache Hive
Apache Impala
Microsoft power bi
Cloudera ODBC

Schematics

IOT Device - Prototype circuit
The prototype rasperry Pi circuit used to simulate data for IN/OUT passengers and bus's door open/close states
Get Me There - Circuit Diag.fzz

Code

busDataSetEdit.pyPython
A script to transform the DataSet by added four columns to each output files, the new columns will match the following:
a. Pass_IN: Random number of passengers getting in the bus at each station
b. Pass_Out: random number of passengers getting at from the bus at each station
c. Total_Pass: total number of passengers onboard of the bus
d. timestamp: change the prod time which is Unix time to Data Time
• Assumption: bus maximum on board number of passengers is 60
#! /usr/bin/env python
# Auther: Mohamed Moussa

import csv
import random
import datetime

# define the input file nd output file
businfoInput = open('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_2013_01_07.csv', 'r')
businfoOutput = open('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01_07.csv', 'w')
businfoReader = csv.reader(businfoInput)
businfoWriter = csv.writer(businfoOutput, lineterminator='\n')

#define maximum number of passengers on board
maxTotal = 60
journyId=0
businfoReader.next()

# loop in the file to add four new columns: Passengers IN, Passengers Out, Number of onboard passengers, Daytime
for row in businfoReader:
	# first station & and applied on all new journyes
	if (row[3] != journyId):
		Utime=int(row[0])/1000000
		# convert Unix time into date time
		Dtime=datetime.datetime.utcfromtimestamp(
       			int(Utime)
    			).strftime('%H:%M:%S')
		journyId = row[3]
		#partitionKey & RowKey columns required n order to upload the file to Micosfot Azure table
		PartitionKey=row[3]
		RowKey=row[7]
		firstIn = random.randint(0, maxTotal)
		firstOut = 0
		onBoard= firstIn
		# write output to the output file
       	 	businfoWriter.writerow([PartitionKey, RowKey]+row+[firstIn, firstOut, onBoard, Dtime])
	else:
		# used for all sttions with the same journy
               	passOut = random.randint(0,onBoard)
       	 	passIn = random.randint(0, maxTotal-(onBoard-passOut))
        	onBoard = (onBoard + passIn) - passOut
		#partitionKey & RowKey columns required n order to upload the file to Micosfot Azure table
                PartitionKey=row[3]
                RowKey=row[7]
		Utime=int(row[0])/1000000
       	 	Dtime=datetime.datetime.utcfromtimestamp(
       	         	int(Utime)
       	         	).strftime('%H:%M:%S')
       	 	businfoWriter.writerow([PartitionKey, RowKey]+row+[passIn, passOut, onBoard, Dtime])

#close the opened files
businfoInput.close()
businfoOutput.close()
busjournyStation.pyPython
to extract the longest journeys based on number of stations
#Script to extract list of longest & shortest Journyes by number of stations
#Auther: Mohamed Moussa

from pyspark import SparkContext, SparkConf
import csv
import glob
import os

conf = SparkConf().setAppName('BusinfoJournyByStation')
sc = SparkContext(conf=conf)

for filename in glob.iglob('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01*'): #reads files from local FS
	fn=filename.split("/")
        fullname=fn[6]
	# to read files from local file system instead of Hadoop
	localFileName="file://"+filename

	trackFile= sc.textFile(localFileName)


	#Function to parse the file into tuples of journyID & Bus Station
	def makeTrackFile(line):
	        l= line.split(",")
	      	#lDate=l[4]
	        lJournyId=l[5]
	        lStation=l[9]
	        return (lJournyId,lStation)

	#filter for values that has na!=None, means to remove all records for the stations that the bus didnt stop at
	fileLines= trackFile.map(lambda line: makeTrackFile(line))
	#select Distinct over the journyID, then reduceByKey to get total number of stations, then add the date then sort the output and remove brackets
	totalByStation = fileLines.distinct().map(lambda x: (x[0],1)).reduceByKey(lambda x,y:(x+y)).map(lambda x: (x[1],x[0])).sortByKey(ascending=False).map(lambda (k, v) : "{0}, {1}".format(k, v))

	#Save output to file - local filesystem
	totalByStation.saveAsTextFile('file:/home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/' + fullname)
buscount.pyPython
to extract list of buses and journey ID
#Script to extract list of buses and Journyes 
# Auther: Mohamed Moussa

from pyspark import SparkContext, SparkConf
import csv
import glob
import os

conf = SparkConf().setAppName('BusCount')
sc = SparkContext(conf=conf)

for filename in glob.iglob('/home/cloudera/DublinBuses010113-310113/DataSet/output/businfo_pass_2013_01_*'): #reads files from local FS
	fn=filename.split("/")
        fullname=fn[6]
	# to read files from local file system instead of Hadoop
	localFileName="file://"+filename

	trackFile= sc.textFile(localFileName)

	#Function to parse the file into tuples of journyID & Bus Station
	def makeTrackFile(line):
	        l= line.split(",")
		lJourneyID=l[5]
	        lbusId=l[8]
	        return (lbusId, lJourneyID)

	fileLines= trackFile.map(lambda line: makeTrackFile(line))
	#select Distinct over the BusID
	totalByBuses = fileLines.distinct().map(lambda (k, v) : "{0}, {1}".format(k, v))

	#Save output to file - local filesystem
	totalByBuses.saveAsTextFile('file:/home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/Buses/' + fullname)
	#os.system("for i in `ls /home/cloudera/DublinBuses010113-310113/DataSet/sparkJob/*.csv/p*`; do cat $i >> /home/cloudera/DublinBuses010113-310113/Output/JournyStationFull.csv ; done")

Credits

Mon image
Abdulrahman Elsharqawy

A passionate Maker

Contact
Me
Mohamed Hassan AbdulRahman

Maker, aiming to make people’s life easier and more meaningful, using the latest HW & SW technologies, analysis and research.

Contact
Csc 0277
Ahmed Emam

Maker

Contact
A58a1c74a60a24d1ad2bcd8f84d6bb17
Ahmed Hassan

Hacker ^--^

Contact

Replications

Did you replicate this project? Share it!

I made one

Love this project? Think it could be improved? Tell us what you think!

Give feedback

Comments

Similar projects you might like

Intelligent Community Alerts Beacon [-iCAB-]
Advanced
  • 39
  • 3

A cloud connected alert system that parses local alert feeds ranging from weather to missing persons and provides visual and audio feedback.

Power Lunch
Advanced
  • 66
  • 2

Full instructions

An insulated food container divided into two sections by a thermoelectric cooler module for quickly heating one side and cooling the other.

Power Lunch

5 members

Sigfox Texting with SiPy
Advanced
  • 1,201
  • 8

Full instructions

Send text messages without any SIM card using the SiPy by Pycom and the Sigfox network!

Open Source, Connected Robot Arm
Advanced
  • 978
  • 11

Work in progress

hRobot is an open source, connected, 3D-printable robotic arm, powered by ROS, with 500g lift capacity

Add A Google Home Ligh Indicator for Voice Activated Google
Advanced
  • 475
  • 4

Full instructions

Give your Google Assistant in Google Home on Raspberry Pi a light indicator.

Run Commands on Your Computers With an Old Remote Control
Advanced
  • 365
  • 3

If you have an old remote control laying around, don't throw it away! You can use it to run commands on your computers over the Internet.

ProjectsCommunitiesTopicsContestsLiveAppsBetaFree StoreBlogAdd projectSign up / Login