ó ¢³[Vc@ sãddlmZddlmZmZmZddlmZmZmZm Z ddlm Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lZdd lZdd lZdd lZdd lmZdd lmZdd lj Z!dd l"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(d„Z)ee*ƒZ+e+j,j-dƒe+j,j.ddddƒe)e+ƒZeƒZ/e/j0e+ƒe"j1e+j,dƒZ2eddƒZ3eddƒZ4ejƒd„ƒZ5d„Z6dd„Z7dd d!„Z8d"„Z9d#„Z:d$„Z;d%„Z<e+j=d&ƒe+j=d'ƒd(„ƒƒZ>e+j=d)ƒd*„ƒZ?e+j=d+ƒd,„ƒZ@e+j=d-d.d/d0gƒd1„ƒZAe+j=d2d.d/d0gƒd3„ƒZBd S(4iÿÿÿÿ(tdivision(tFlasktgtsession(trender_templatetflashtredirectturl_for(trequesti(tSiteForm(tSession(tCeleryttask(tMySQL(t SQLAlchemyN(t namedtuple(t itemgetter(tCrawlerProcess(tsolo(tUserInputPipelinec sdtˆjdˆjdƒ}|jjˆjƒ|j‰dˆf‡‡fd†ƒY}||_|S(sT Init celery environment with Redis broker. Allows use of celery decorator. tbrokertCELERY_BROKER_URLt ContextTaskc s eZeZ‡‡fd†ZRS(c s*ˆjƒˆj|||ŽSWdQXdS(N(t app_contextt__call__(tselftargstkwargs(tTaskBasetapp(sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyR)s (t__name__t __module__tTruetabstractR((RR(sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyR's(R t import_nametconfigtconftupdatetTask(RtceleryR((RRsX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyt make_celerys   R#Rsredis://localhost:6379tCELERY_RESULT_BACKENDtCORTIPY_API_KEYtSites siteurl textt CorticalSites!siteurl text fingerprint keywordscC sZtidd6idd6d6dd6id d6d 6ƒ}|jtjd |ƒ|jƒd S( sZ Initializes Scrapy crawler for given siteurl. See spider/pipeline code for more. s2Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)t USER_AGENTids1app.vcspider.vcspider.pipelines.UserInputPipelinetITEM_PIPELINESit DEPTH_LIMITts3tDOWNLOAD_HANDLERStdomainN(RtNonetcrawlRt SoloSpidertstart(tsiteurltprocess((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytscrapeEsc C sUtjƒ}|jƒ}d}|j||fƒ|jƒ}|jƒ|dkrÙtdƒd}|j||fƒ|jƒ\}}}}|jƒtj |ƒ}t |t ƒrÃ|d}nt ||||gƒStdj |ƒƒtj|ƒ}xF|dkrC|j||fƒ|jƒ}|jƒtjdƒqþWt||ƒSdS(sM Either scrapes text of input site, or returns already-scraped data. s7SELECT text FROM crunchbase_startups WHERE siteurl = %ss1Site already scraped, proceeding with analysis...stSELECT siteurl, text, cortical_io, cortical_io_keywords FROM crunchbase_startups WHERE siteurl = %st positionssTrying to scrape site: {}iN(tmysqltconnecttcursortexecutetfetchonetcommitR3Rtastt literal_evalt isinstancetdicttCSitetformatR9tdelayttimetsleepR+( R7tcontcurtsqlttextt_t fingerprinttkeywordstscraper((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytget_siteTs0          teuclideanDistancecC stj|j|jƒ|S(s+ Now just for QA, get API distance (tclienttcompareRO(tsite1tsite2tmetric((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyt getSDRDist}st TextInputicC såtj||dgdƒ}tj|dƒ}tjƒ}|jƒ}|dkr„d}|j|t|ƒdj|ƒ|fƒn=|dkrÁd}|j||t|ƒdj|ƒfƒn|j ƒt |||d|gƒS(sF Makes an SDR and associated keywords for input site or text. its~UPDATE crunchbase_startups SET cortical_io = %s, cortical_io_keywords = %s WHERE siteurl = %st,isjINSERT INTO sitedescriptions (text, cortical_io, cortical_io_keywords) VALUES (%s, %s, %s)R:( RTtcreateClassificationtextractKeywordsR;R<R=R>tstrtjoinR@RE(RMR7tisTexttsite_corticalmapt site_keywordsRJRKRL((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytmakeSDR„s   + + cC sEtjƒ}|jƒ}d}|j|ƒ|jƒ|jƒ}|S(s/Get all VCs for comparison to selected startup.s¸SELECT siteurl, cortical_io, cortical_io_keywords FROM vctest4 WHERE NULLIF(text, '') IS NOT NULL AND NULLIF(cortical_io, '') IS NOT NULL ORDER BY RAND()(R;R<R=R>R@tfetchall(RJRKRLtvcList((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyt loadVCList¡s     cC s:t|ƒt|ƒ}tt|ƒt|ƒAƒ}||S(sThe Euclidean distance is defined as a float between 0 and 1, 0 representing a smaller distance and thus closer match between two SDRs. It is calculated as the quotient of the length of the symmetric difference between two sets and the total length of the combined sets.(tlentset(tl1tl2ttotlentsublen((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyteucDist·scC sCtt|ƒt|ƒ@ƒ}t|ƒt|ƒ}|tj|ƒS(sHere (and only here, per cortical.io), cosine similarity is defined as a float between 0 and 1, 1 representing a high similarity. It is calculated as the quotient of the length of the number of bits contained in both sets and the square root of len(l1) * len(l2).(RhRitnptsqrt(RjRktoverlapttotsize((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytcosSimÃscC stƒ}tƒtƒ}}x†|D]~}t|jtj|dƒƒ}|jt|d|gƒƒt|jtj|dƒƒ}|jt|d|gƒƒq#Wt |dt dƒƒ}t |dt dƒdt ƒ}t dj |d ƒƒt dj |d ƒƒt||fƒS(sàFind top three matches for the given startup based on both Euclidean distance and Cosine similarity (calculated per cortical's API, not really mappable to traditional definitions, ESPECIALLY cosine similarity.) iitkeytreverses Euclidean: {}is Cosine: {}(RgtlistRnRORARBtappendttupleRstsortedRR RRF(tstartupRft scoresEuct scoresCostvctscoreEuctscoreCos((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytgetMatchÎs  !t/s/indexc C s"tdddddddddƒS( Ns index.htmlttitletHometh1tactiveth2tinactiveth3(R(((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytindexìs  s/aboutc C s"tdddddddddƒS( Ns about.htmlR‚tAboutR„R‡R†R…Rˆ(R(((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytaboutõs  s/contactc C s"tdddddddddƒS( Ns contact.htmlR‚tContactR„R‡R†RˆR…(R(((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytcontactýs  s/processtmethodstGETtPOSTc C sötjjdƒ}|dkrNtjjdƒ}tt|fƒddƒ}nT|dkr¢tjjdƒ}t|ƒ}t|tƒr¢t|j |j ƒ}q¢nt dj |j |j ƒƒt dƒt|ƒ}tdd d d d d d dd ƒS(Ntdescrtyt siteinputRaisKeywords for site {}: {}sGetting best match...s process.htmlR‚tResultsR„R‡R†Rˆ(RRtgetRdRxR3RRRCR+RMR7RRFRPR€R(R‘R“tsitedataR7tmatches((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pyR8s      s/inputc C sbttjƒ}tjdkr:|jƒr:ttdƒƒStdddddddd dd |ƒS( NRR8s input.htmlR‚s Enter siteR„R‡R†Rˆtform(R RR˜tmethodtvalidateRRR(R˜((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytinput s (Ct __future__RtflaskRRRRRRRRtformsR tflask.ext.sessionR R'R R tflask.ext.mysqlR tflask.ext.sqlalchemyRtreRARHtnumpyRot collectionsRtoperatorRtmysql.connectort connectortmsctcortipytscrapy.crawlerRtvcspider.vcspider.spidersRtvcspider.vcspider.pipelinesRR(RRR#t from_objectR%R;tinit_apptCorticalClientRTR+RER9RRRYRdRgRnRsR€trouteR‰R‹RR8R›(((sX/Users/johnmontroy/Documents/Learning/NYC-DSA/Project 3/banker.ai/mcubed/app/__init__.pytsZ"            )    $