Price Parser

Currently, for different online retail sites, we hard coded the html tags that the parser should look for to get product information. For the whole term, I have been wondering if there is a better way to do it. Once the sites change their UI design, we'll have to change our code. After doing research online and talk to different people, I think that's just the nature of the html parsing. Other than using the retail site's API (if it exists), hard coding the parsing info seems to be the correct way to use! 

Last Few Updates

As the deadline nears, here are a few other additional features that we added since the Demo Day.

• Give user an option to choose when to stop notifying them by
• There is a widget where the user selects when they want to stop tracking the item
• Default is None
• Show a list of the same item but from different stores
• This is useful for users who are curious where they can find the same item but cheaper

Latest Product Page:

Current Logo

After many weeks, we have decided on the following logo for now:

Demo Day + More Updates

Demo Day

Despite not making the Top 3, Demo Day went pleasantly well on Thursday. We received plenty of compliments and several people gave us their email once we finalize our website. As it turns out, everyone shops online and has at one point in their life wished they could be automatically notified about price drops. They also liked the simplicity of our website and ease of use. One person, who owns an Italian website in charge of notifying users of coupons for stores, offered to talk to us about a potential partnership. Interesting.

In addition to compliments, we got several constructive criticisms that we will consider implementing in the near future:
- Giving users an option to set a day to stop notifying them by
- In the item details page, show a list of the same item in competitive stores and their prices
- Taking advantage of Amazon's Recommendation items
- Prettify the price history in graph form


Mini-Updates
- Added track/untrack buttons
- Notification emails contain information about item drop (such as new price) and contains link to send users to the actual website

SAMPLE EMAIL

FUTURE CONSIDERATIONS:
- Same as the last post plus the suggestions made by people from Demo Day
- Perhaps create a history page of user's past tracked items.
- Create a button for users saying that they purchased that item

• After a purchase, we still want to track that item for a certain time because some stores have a Buy Back policy

Long-term goal: Develop a store credit system similar to Ebay where buying through our website gives users store credit to buy other products. Must be partnered with other companies to work

Update So Far

As the demo day nears, here are some of the things we've accomplished recently and hope to accomplish in the near future:

Added a detailed products page for each item

• Clicking on an item sends user to a page showing details about that item
• Displays item information
• Displays price history
• Displays recommendations

Added tags to each item for our recommendation system

• Currently uses product names for tags
• Recommendations then uses those tags to decide which items to recommend
• Clicking on a tag lists all items with the same tag

Incorporate price history

• Display date and price in list form

Add commenting system

• Users can comment on a specific item for everyone to view

Added a few more links on each page so user can more easily navigate website

Agreed upon a logo

FUTURE CONSIDERATIONS

• Research smarter recommendation algorithms
• Graphically display price history
• Visually enhance each page
• Give more options to users
• Add track/untrack buttons
• Make emails more meaningful
• In addition to recommendations page, possibly add a "Users who track this item also track..." section

Week 6 Update

Successfully deployed on Fastsoft's EC2

• http://131.215.221.146/mypage/
• Incorporated Lucia's css designs

CURRENT WELCOME PAGE

List of tasks still to be done:

• Work on sending email confirmations
• Redesign user home page to include more functionality
• Site map
• If enough time, work on graphs and displaying statistics
• Register domain name

Week 5 Update

In addition to the progress as described by the posts below...

Created a user's My Page with the following features
1. Entering a product url to be tracked can now be done from the homepage
2. Lists all currently watched items
   i. Each item has current price, store name, id, picture, and last price change

CURRENT MY PAGE

URL Enter work flow

When user insert a URL to keep track:
 - parse the url
 - insert/update in products_item (do select on product_id and store)
   - if exists already in products_item, update the information.
   - else insert into products_item
 - insert into products_tracklist

 pseudocode
 * parse the information
 * select id, price from products_item where product_id  = blah and store = blah;
 * if (price == null)
 *    insert into products_item values (info);
 *      insert into price_history the new price
 * else if (price != new_price)
 *    update products_item set price = new_price, price_date = current timestamp where id = id;
 *    insert into price_history (item_id, price) values (id, new_price);
 * insert into products_tracklist if it does not exist yet.

Crawl....

Completed the following:

• Given a starting URL and url filter format (which help to keep inside of the website domain), a multithreaded crawler explores inside a target website. 
• For each website:
• It parses the price.
• Store inside of the database using connection from database connection pool. The database maintain a set of connection that is ready to use.
• Add the outlink to the queue that is going to be visit.
• Set up github in the cloud
• The code is in the server already. 

To Do:

• There is bug when the crawler finished, which seems to come from multiple processes not stopping correctly.
• Currently, the crawler is only for one website. We need to write a simple for loop to make it automatically crawler a set of website domains.

Week 4 Progress

Currently working on:

• User account page.
• Display the list of item a user has. For each item, we'll display the picture of the item, old price, new price, item name, and item store id.
• Set up mysql database on the cloud.
• Change from sqlite3 to mysql, and create connection pool to insert faster.
• Write a crawler to visit websites and parse the prices.

User Registration and Price Parsing Strategy

Implement completed user registration framework, including login, logout, password reset, and register. The old price parsing strategy is to check the URL for updates, but we realized that a same item have different URL before and during sale. The new strategy: 1. For website with API, such as Amazon and Bestbuy, we will use that to retrieve to price. 2. For relative small website, such as jcrew and express, we will crawl the whole website and update the item database based on store product id.

Week 3 Progress

Completed the following:

1. Created url page to ask user for item url
2. Run my Python program to parse url for essential information and store into database
1. Information includes productName, productId, productPrice, storeName, date
3. Wrote Python script to traverse database and look for any changes in price for each item
1. Cronjob will run this script every time interval (1 hour)

To do:

1. Use Amazon/Bestbuy API as a more efficient alternative to look for any updates/changes to prices
2. Cover edge cases not caught by the parsing program
3. Instead of using url to look for updates, since urls can change, find better alternative
4. Combine with Lucia's frontend design
1. Right now my url page is standalone, want to combine with user 's main page

Making Logos

Now that we came up with our website name, I decided to make a logo for it. I found a very nice font called, "Alba," created by Fontalicious from dafont.com and used it. Most of the work is done in photoshop. Here are the logos I made.
Big Rainbow:                                  

Small Rainbow:

Lighter Rainbow:

Watercolor 1:

Watercolor 2:

Tuling does not like the logos with rainbow :'(
Which one do you guys like the most?

Week 2 Progess

1. Successfully scraped the following sites for product price, name, and id:
    Amazon, BestBuy, Express, Forever21, Walmart, Target

Week 1 Progress (UPDATED)

1. Set up a simple website using Amazon Web Services
     a. Running Apache on an Amazon EC2 Instance using Amazon Linux AMI
     b. Cheaper and more convenient than setting up web server on our own machines

2. Successfully take in user input and store information in a database

3. Came up with name
     a. onomnom

Projected Timeline

Timeline

Week 1 (user interface and be able to send information to backend)

●     Set up website on Amazon Web Services

○     Cheap, scalable

○     Use of Amazon Simple Storage Service for storing data

●     Be able to take in user input on the backend

○     URL of the item

○     Email address

●     Set up the basic user interface

●     Come up with a name.

Week 2 - 5 (backend)

●     Be able to parse the price from most of the websites

○     Write scripts incorporating regular expressions to read prices from HTML files

○     Research how to frequently check if the price is changed.

●     Store parsed data in a database

○     Design tables to keep track of product id, url, price

○     Write functions to query and update tables

●     Test the backend to make sure it works accurately and properly.

Week 6 (send information from backend to user)

●     Work on email notifications

●     Notify user when price drops or drops below a certain price

Week 7 - 9 (add more functionality of the website)

●     Also allow user create an account

●     Aggregate data in a form that is friendly to the user

○     Let user see the history of requests he made.

○     Let user see the price history of the watched item in form of a graph

○     Allow user to sort the information in different ways, for example, he can sort by lowest price, or most recently changed price.

○     Allow user to put different items to different “baskets”, for example, “Purchased”, “Gifts”, “Wish List”.

●     Suggest related items to watched item

●     Suggest related websites with same items

Week 10 (frontend)

●     Improve on UI design

●     More testing.

Introduction

Every shopper loves to buy products at a bargain, but hates the effort of searching and shopping around for the best deal.  Bargain hunting websites do exist, but still require some work to hunt down the desired product.  We hope to create a website that gives the savvy shopper a way to track the price of a specific product without the hassle of surfing the web every day.  The user can enter the URL of some product and then receive an alert if the price on that page drops and is also shown retailers who carry the product or similar products.  Even after the user purchases the product, the website will still check for price drops for a few days so that should the price continue falling, the user can take advantage of the price exchange policy to get an even better deal.

This type of website targets the “lazy” shopper who does not want to constantly check for sales since this website will alert the user via e-mail.  Since many similar shopping-assistant websites already exist, like Groupon, Ebates, and Price Protectr, with similar functions, this idea may not survive as a startup.  However, this website will be unique in that it takes the best features of many shopping-assistant sites, like the price tracker and recommendation system, to help users shop for their desired products in the cheapest possible way.  Some of these useful features include alerting the user of price drops, recommending retailers where a lower price may be offered, organizing products that the user may want to purchase, and offering better deals for products bought through the website.

This website will have three key functions.  The first is to take as an input the URL to a specific product and alert the user of any price drop that occurs as soon as possible.  The second is to recommend similar retailers who also sell the product.  The purpose of the recommendation list is not to show lower prices from other retailers, but rather to give the user alternative places to purchase the product.  For example, the list may just be called, “These retailers also sell this product:”.   Finally, another recommendation list will show products similar to the specific product as done on any retailer website.  Since we will have a history of products that users are interested in, we can implement this feature using algorithms similar to Amazon’s “Frequently Bought Together” and “Customers Who Bought This Item Also Viewed” functions.

All of these features are meant to assist the user in finding the best deal on a certain product or similar product by alerting them of falling prices both before and after the purchase and of displaying options of where to purchase the product.  Therefore, this website needs to check the given URL for updated prices as often as possible so that the user can be alerted quickly to take advantage of a sale or possibly diminishing supply of the product.

To simplify the development of this website, we will start by supporting only one or two input websites (for example the URL given by users must be from a specific store or two).  Once the functionality for the features is up and running for these starting retailers, the website can expand to take URLs from other retailers’ websites as well.

Related Websites

The following are similar websites and their descriptions online.

Price Protectr - http://www.priceprotectr.com/

Use Price Protectr to track prices of things you're thinking about buying, and save money even after you buy. There are lots of stores out there that offer price protection policies -- when the price drops on an item you've purchased, they'll refund you the difference. But there's a catch... it's up to you to watch prices. Price Protectr makes it simple to watch prices, keep track of your purchases, and get rebates off price drops.

Camelcamelcamel - http://camelcamelcamel.com/

Camelcamelcamel provides free Amazon price drop alerts and Amazon price history charts, helping users buy when the price is right.

Become - http://www.become.com/

Online shoppers love finding exactly what they want online and we love providing them with the comparison shopping technology and large selection of products to do it! Save time and money through Become.com’s comparison shopping site and shop for online discounts and the lowest prices from name brand retailers. From diamond rings to smart phones, find the best online deals and interesting shopping insights on Become.com!

Our shoppers can also depend on Become.com’s product research engine for advice on what they’re shopping for online, from buying guides for LCD televisions to product reviews on baby strollers, as well as editorial tips for the hottest online deals and products for the season. Use our comparison shopping site to find your next purchase and give us your feedback on how to make our online shopping experience even better!

Become does not have our price tracking feature, but it does have a great price comparison feature when a user searches for an item that we would like to implement in our “suggested similar items” section. Meanwhile, Camelcamelcamel has a great feature of showing the history of prices, but since it only works with a few websites such as Amazon, Best Buy, Newegg, etc, we wish to build on its price history line graph to provide users with more options. Overall, the website most similar to our idea is Price Protectr. Price Protectr provides services that watch prices of items in online stores, alerts when the price of the products drop, and deals with refund issues when users buy products before their prices decrease by giving detailed information about what users should do to get money back. What separates our website from Price Protectr and camelcamelcamel is our desire to build a better user interface, and provide two additional recommendation features, discussed below.

General Approach Based on Prior Work Done in Class

Our website will have two recommendation features. One feature is to suggest different websites that sell the same item that a user wants to buy. Each website has a different price and deals so we want to provide the user with more options to get the best deal available. For example, if a user wants to buy a Macbook Pro, he/she can choose from various online stores such as Apple, Amazon, or Best Buy. Our website will crawl these websites and show our users everything he/she wants to know without requiring them to visit multiple sites at once. Another recommendation feature that we plan to include will suggest items similar to the item that the user wants to purchase. If a user does not have a strong preference to buy the item he/she originally intends to buy, we might be able to sway the user into buying a similar item. This feature will be similar to Amazon’s “What Other Items Do Customers Buy After Viewing This Item?” list.

We will create a web crawler to constantly browse the items’ webpages that our users submit for price watch and to parse the prices for any updates to the price. In Homework 2, we already wrote a crawler that traverses through websites within Caltech’s domain. For our website, users will input the address of the webpages they want us to watch over. Then our crawler will take those inputs, visit the webpage, and send relevant information back to our website.

To enable the feature of recommending similar items, we will study and utilize the ideas of the web graph and characteristics like clustering. We also need to decide how often we are crawling different websites to get the price, which involves studying the heavy tail distribution to figure out which websites need to be crawled more frequently. Since we are keeping a list of desired products of each user after they input an URL, we can study how successful we are at the “User who viewed this also viewed” section. The results we collect will be used to constantly improve the features and functionality of the website.

Our website will not only search for similar items but also enable sellers to partake in the suggestion process. We will create ad positions for suggestions and let sellers or advertisers bid on these ad positions to compete for the opportunity to advertise their products. Before opening up the bid, we require that the items of each advertiser follow a certain criteria; we do not want unrelated items spamming the ad space of a certain product. Then, based on the bids, we will decide when and where to show the products and prioritize certain items if the bidders of those items bid higher than their competitors. 

Product

The end product will be a website that assists users with their personal shopping experience.  Its functions are:

1.    Given a URL of a product, the website would inform users via email once the price has dropped. The alerts will continue until a few days after the user makes a purchase so that if the price continues falling the user can take advantage of the price exchange policy to get an even better deal.

2.    The website will keep a product price history available so users can look up how similar products’ prices have changed and decide when is the best time to make a purchase.

3.    The website will suggest other places to buy the same item, especially if a cheaper price can be found. We will also use this feature for situations when the desired product is sold out on the original website.  Positions in this list will be determined by auction.  Retailers can bid for the highest positions on certain products.The purpose of this is to give users an alternative place to shop.

4.    To make a related products list, we can look at shopping behavior and recommending products algorithms, similar to Amazon’s “Frequently Bought Together” and “Customers Who Bought This Item Also Views” sections.  The purpose of this is to encourage the user to continue shopping by recommending similar products.

5.    Finally, we would ideally like to pair up with companies and give people money back for purchasing products through the website. This money could be in form of store credit that they can use to buy more products through our website, similarly to eBay’s eBay Bucks, or it could be checks or gift cards, similarly to Ebates’ cashback. (The point will not be accomplished in this semester).

References

“Tutorial: Develop a local Amazon price tracking application.”

http://www.yuonlamp.com/local_amazon_price_watcher.html

http://www.priceprotectr.com/

http://camelcamelcamel.com/

http://www.become.com/