CTPP

ctpp@listserv.transportation.org

5 participants
3069 discussions

[CTPP] Breaking News! SHP Files downloadable!

by Weinberger Penelope

Hi All, As many of you know, we have been having a problem with intermittent availability of SHP file downloads. This appears to be fixed. Penelope Z. Weinberger CTPP Program Manager AASHTO 202-624-3556 ctpp.transportation.org

11 years, 8 months

[CTPP] CTPP 2006-2010 FAQ is available on-line!

by Liang Long

Dear CTPP Users: Are you wondering what are the differences between CTPP 2006-2010 and Standard ACS products? Are you struggling Set A and Set B tables from the recent released CTPP? If so, please check our new FAQs on the FHWA website: http://www.fhwa.dot.gov/planning/census_issues/ctpp/faq/index.cfm As always, if you have any questions on the CTPP, feel free to contact me. Liang Long Cambridge Systematics, Inc. 4800 Hampden Lane Suite 800 Bethesda, MD 20814 tel 301 347 0100 fax 301 347 0101 FHWA 202-366-6971 e-mail llong(a)camsys.com www.camsys.com

11 years, 8 months

[CTPP] FW: Adventures in large datasets

by Weinberger Penelope

From: Howard Slavin [mailto:howard(a)caliper.com] Sent: Friday, November 15, 2013 11:32 AM To: 'ctpp-news(a)chrispy.net' Subject: RE: [CTPP] Adventures in large datasets Thanks to Krishnan for the kind mention of TransCAD. I wanted to let everyone know that Caliper is now distributing the ctpp flow data in matrix form for TransCAD users. While the data have been provided at the Transportation Analysis Zone level, only the Census tract to Census tract flow data are complete, and these are the data that Caliper is now providing to TransCAD users in the United States. The flow data are provided in the form of two TransCAD matrices that provide the estimated flows between all of the Census tracts in the U.S. and that distinguish estimated work trip flows with two different breakdowns by means of transportation and margins of error. The dimensions of these matrices are roughly 70,000 by 70,000, but remarkably they compress to only 1.5 GB or less and so are downloadable. With TransCAD, it is easy for users to subset the matrices to cover any geographic subarea of interest. These files can be downloaded by users at http://www.caliper.com/Press/census-ctpp-worker-flow-data-download.htm We expect to make the TAZ to TAZ flow data available as soon as it is corrected by the Census Bureau. Howard From: ctpp-news-bounces(a)chrispy.net<mailto:ctpp-news-bounces(a)chrispy.net> [mailto:ctpp-news-bounces(a)chrispy.net] On Behalf Of Krishnan Viswanathan Sent: Thursday, November 14, 2013 9:59 PM To: ctpp-news(a)chrispy.net<mailto:ctpp-news(a)chrispy.net> Subject: Re: [CTPP] Adventures in large datasets Mara Besides SQL server I have the following suggestions: 1) the ff package in R ( http://www.bnosac.be/index.php/blog/22-if-you-are-into-large-data-and-work-…) 2) HDF5 seems like a decent option though I have not used it. Link to rhdf5 ( http://bioconductor.org/packages/release/bioc/html/rhdf5.html). Also, SFCTA has some code for getting data into and out of HDF5 ( https://github.com/sfcta/TAutils/tree/master/hdf5) 3) I have found TransCAD to be efficient in processing large datasets. Hope this helps. Krishnan I downloaded the Maryland state raw data (the whole enchilada) that Penelope was good enough to provide me. It came with documentation that clearly explains what needs to be done but I am being hampered by the sheer size of the dataset. It's 10 GB and that's without going into joining tables, transposing them to meet my needs, etc. Even breaking the parts into different databases it can't be handled in Access. I can fit Part 1 into an ESRI geodatabase but I don't have the flexibility in linking tables that Access has. Does anyone have any suggestions for dealing with large databases? SQL server is one option. Are there others? Mara Kaminowitz, GISP GIS Coordinator ......................................................................... Baltimore Metropolitan Council Offices @ McHenry Row 1500 Whetstone Way Suite 300 Baltimore, MD 21230 410-732-0500 ext. 1030<tel:410-732-0500%20ext.%201030> mkaminowitz(a)baltometro.org<mailto:mkaminowitz(a)baltometro.org> www.baltometro.org <http://www.baltometro.org/> _______________________________________________ ctpp-news mailing list ctpp-news(a)ryoko.chrispy.net<mailto:ctpp-news(a)ryoko.chrispy.net> http://ryoko.chrispy.net/mailman/listinfo/ctpp-news

11 years, 9 months

Re: [CTPP] CTPP 2006-2010 Software Cheatsheets

by Tom Faella

Dear CTPP listservers: These are very helpful! Thank you Liang, Tom Faella Executive Director, La Crosse Area Planning Committee (LAPC) La Crosse County Administrative Center 400 4th Street North, Room 2300 La Crosse, WI 54601 (608) 785-5977 (608) 793-6525 (Fax) www.LAPC.org tfaella(a)lacrossecounty.org<mailto:tfaella(a)lacrossecounty.org> From: ctpp-news-bounces(a)chrispy.net [mailto:ctpp-news-bounces(a)chrispy.net] On Behalf Of Liang Long Sent: Wednesday, November 13, 2013 3:00 PM To: ctpp-news(a)chrispy.net Subject: [CTPP] CTPP 2006-2010 Software Cheatsheets Dear CTPP Users: Please find attached the CTPP 2006-2010 software cheat sheets that we have developed. For example, if you are trying to get tract to tract or TAZ to TAZ flows for multiple counties, you may find the cheat sheet for part 3 useful. Feel free to contact me if you have any questions. Liang Long Cambridge Systematics, Inc. 4800 Hampden Lane Suite 800 Bethesda, MD 20814 tel 301 347 0100 fax 301 347 0101 FHWA 202-366-6971 e-mail llong(a)camsys.com<mailto:llong(a)camsys.com> www.camsys.com PRIVATE AND CONFIDENTIAL This e-mail and attachments are intended for the addressed recipient only. If you are not the correct recipient please notify the sender of the delivery error and delete this message. Improper disclosure, copying, distribution, retransmission, or use of information from this e-mail is Prohibited, and may result in liability and damages for misuse of this information.

11 years, 9 months

[CTPP] Fwd: Call for abstracts for the ACS Data Users Conference

by Ed Christopher

This may be of interest to some. > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > > ACS Data Users Conference > > May 29-30, 2014 > Holiday Inn Capitol, Washington D.C. > > Call for Abstracts! > Deadline: January 15 > Notification Date: February 15 > > To submit an abstract, visit www.acsdatausers.org. The $200 conference registration fee will be waived for persons presenting at the meeting. > > The American Community Survey (ACS) Data Users Conference will bring together ACS data users and staff from the U.S. Census Bureau to improve understanding of the value and utility of ACS data and to promote information sharing among data users about key ACS data issues and applications. The conference will include a mix of invited and contributed presentations by ACS data users and Census Bureau staff. > > We are inviting abstract submissions on the following topics (submissions on other applications of ACS data are also welcome): > Using the ACS to look at trends over time, including multiyear estimates > Issues and limitations in using small-area ACS data > How data users have dealt with changes in ACS survey questions > Aggregating ACS estimates and calculating margins of error > Using ACS in GIS and other mapping applications > How to access ACS from different sources > Applications of ACS data in different topical areas: Health insurance and disability; Income and poverty; employment and workforce development; and Migration patterns and trends > Please forward this announcement to others who may be interested. More information about the conference, including registration information, will be available in the coming weeks. > >

11 years, 9 months

Re: [CTPP] Adventures in large datasets

by Krishnan Viswanathan

Hary Thanks for the clarification and pointing out the drawbacks with using HDF5. I am all for open source software so if MySQL etc. works then great. I found TransCAD really fast in processing large (though not using 10 GB) datasets esp. when sorting and found it much faster than standard statistical packages. Krishnan On Fri, Nov 15, 2013 at 9:03 AM, hprawiranata mitcrpc.org < hprawiranata(a)mitcrpc.org> wrote: > Krishnan, > > HDF was data format, all climate data is stored on HDF format (I > experience writing C code for this data format on SPARC computers long > time ago and from HDF I converted to MySQL ! ). CTPP is just a flat > data set, txt. Converting into HDF is a different complicated story > and on a wrong direction. > > Linking data tables from many tables has to be done using database > engine. MS Access won't cut it. M$ SQL is expensive. Free, Open Source > is the only way and fast. > > TransCAD... I never test the limit. But linking simple tables is ok. > Large and many tables ? Better use data modeler software and link to > database engine. > > Hary(ono) Prawiranata > Transportation Analyst/Modeler > Tri-County Regional Planning Commission > 3135 Pine Tree Rd. Ste 2C > Lansing, MI 48911 > > On Thu, Nov 14, 2013 at 9:58 PM, Krishnan Viswanathan > <krisviswanathan(a)gmail.com> wrote: > > Mara > > > > Besides SQL server I have the following suggestions: > > 1) the ff package in R ( > > > http://www.bnosac.be/index.php/blog/22-if-you-are-into-large-data-and-work-… > ) > > 2) HDF5 seems like a decent option though I have not used it. Link to > rhdf5 > > ( http://bioconductor.org/packages/release/bioc/html/rhdf5.html). Also, > > SFCTA has some code for getting data into and out of HDF5 ( > > https://github.com/sfcta/TAutils/tree/master/hdf5) > > 3) I have found TransCAD to be efficient in processing large datasets. > > > > Hope this helps. > > > > Krishnan > > > > I downloaded the Maryland state raw data (the whole enchilada) that > Penelope > > was good enough to provide me. It came with documentation that clearly > > explains what needs to be done but I am being hampered by the sheer size > of > > the dataset. It's 10 GB and that's without going into joining tables, > > transposing them to meet my needs, etc. Even breaking the parts into > > different databases it can't be handled in Access. I can fit Part 1 > into an > > ESRI geodatabase but I don't have the flexibility in linking tables that > > Access has. > > > > > > > > Does anyone have any suggestions for dealing with large databases? SQL > > server is one option. Are there others? > > > > > > > > Mara Kaminowitz, GISP > > GIS Coordinator > > ......................................................................... > > Baltimore Metropolitan Council > > Offices @ McHenry Row > > 1500 Whetstone Way > > Suite 300 > > Baltimore, MD 21230 > > 410-732-0500 ext. 1030 > > mkaminowitz(a)baltometro.org > > www.baltometro.org > > > > > > > > > > _______________________________________________ > > ctpp-news mailing list > > ctpp-news(a)ryoko.chrispy.net > > http://ryoko.chrispy.net/mailman/listinfo/ctpp-news > > > > > > _______________________________________________ > > ctpp-news mailing list > > ctpp-news(a)ryoko.chrispy.net > > http://ryoko.chrispy.net/mailman/listinfo/ctpp-news > > > _______________________________________________ > ctpp-news mailing list > ctpp-news(a)ryoko.chrispy.net > http://ryoko.chrispy.net/mailman/listinfo/ctpp-news > -- Krishnan Viswanathan 5628 Burnside Circle Tallahassee FL 32312

11 years, 9 months

[CTPP] FW: Call for Abstracts: ACS Data Users Conference

by Weinberger Penelope

fyi From: ACS Data Users Group [mailto:ACS_Data_Users_Group(a)mail.vresp.com] Sent: Friday, November 15, 2013 3:01 PM To: Weinberger Penelope Subject: Call for Abstracts: ACS Data Users Conference [ACS data users logo] ACS Data Users Conference May 29-30, 2014 Holiday Inn Capitol, Washington D.C. Call for Abstracts! Deadline: January 15 Notification Date: February 15 To submit an abstract, visit www.acsdatausers.org.<http://cts.vresp.com/c/?PopulationReferenceB/d13a469f3e/2af176bb3e/c961d350…> The $200 conference registration fee will be waived for persons presenting at the meeting. The American Community Survey (ACS) Data Users Conference will bring together ACS data users and staff from the U.S. Census Bureau to improve understanding of the value and utility of ACS data and to promote information sharing among data users about key ACS data issues and applications. The conference will include a mix of invited and contributed presentations by ACS data users and Census Bureau staff. We are inviting abstract submissions on the following topics (submissions on other applications of ACS data are also welcome): * Using the ACS to look at trends over time, including multiyear estimates * Issues and limitations in using small-area ACS data * How data users have dealt with changes in ACS survey questions * Aggregating ACS estimates and calculating margins of error * Using ACS in GIS and other mapping applications * How to access ACS from different sources * Applications of ACS data in different topical areas: Health insurance and disability; Income and poverty; employment and workforce development; and Migration patterns and trends Please forward this announcement to others who may be interested. More information about the conference, including registration information, will be available in the coming weeks. ________________________________ Click to view this email in a browser<http://hosted.verticalresponse.com/1056151/d13a469f3e/521474375/2af176bb3e/> If you no longer wish to receive these emails, please reply to this message with "Unsubscribe" in the subject line or simply click on the following link: Unsubscribe<http://cts.vresp.com/u?d13a469f3e/2af176bb3e/mlpftw> Click here<http://oi.vresp.com/f2af/v4/send_to_friend.html?ch=d13a469f3e&lid=521474375…> to forward this email to a friend Population Reference Bureau 1875 Connecticut Ave., NW Suite 520 Washington, District of Columbia 20009 USA Read<http://www.verticalresponse.com/content/pm_policy.html> the VerticalResponse marketing policy. [Try Email Marketing with VerticalResponse!]<http://www.verticalresponse.com/landing/ef/?mm/d13a469f3e&utm_campaign=foot…> [http://cts.vresp.com/o.gif?d13a469f3e/2af176bb3e/mlpftw]

11 years, 9 months

Re: [CTPP] Adventures in large datasets

by hprawiranata mitcrpc.org

Actually, I have downloaded whole Michigan CTPP flat file ! Now, in extracting process. processing it, total data 15,523 MB !, I will dump it into MySQL and play with data modeler tools I have. Thanks ! Hary(ono) Prawiranata Transportation Analyst/Modeler Tri-County Regional Planning Commission 3135 Pine Tree Rd. Ste 2C Lansing, MI 48911 On Fri, Nov 15, 2013 at 8:52 AM, hprawiranata mitcrpc.org <hprawiranata(a)mitcrpc.org> wrote: > My experience handling large data set (climate data set, on linux, > with parallel computers, and with R, Python) was using free software > MySQL (or PostgreSQL, mostly I use MySQL), for easy access and > management use MySQL workbench ( > http://www.mysql.com/products/workbench/ ). > If you already install MOVES 2010, you can use MOVES MySQL engine to > dump your data. I use this MySQL to process vehicle registration data > (~9 million records of Michigan vehicles), it is faster than MS Access > cannot handle it.And it is on Windows.. (I have another version on > Linux, too). > > I'm interested to dump Michigan data too, how big is this ? Penelope ? > > Hary(ono) Prawiranata > Transportation Analyst/Modeler > Tri-County Regional Planning Commission > 3135 Pine Tree Rd. Ste 2C > Lansing, MI 48911 > > > On Thu, Nov 14, 2013 at 2:19 PM, Mara Kaminowitz > <mkaminowitz(a)baltometro.org> wrote: >> I downloaded the Maryland state raw data (the whole enchilada) that Penelope >> was good enough to provide me. It came with documentation that clearly >> explains what needs to be done but I am being hampered by the sheer size of >> the dataset. It's 10 GB and that's without going into joining tables, >> transposing them to meet my needs, etc. Even breaking the parts into >> different databases it can't be handled in Access. I can fit Part 1 into an >> ESRI geodatabase but I don't have the flexibility in linking tables that >> Access has. >> >> >> >> Does anyone have any suggestions for dealing with large databases? SQL >> server is one option. Are there others? >> >> >> >> Mara Kaminowitz, GISP >> GIS Coordinator >> ......................................................................... >> Baltimore Metropolitan Council >> Offices @ McHenry Row >> 1500 Whetstone Way >> Suite 300 >> Baltimore, MD 21230 >> 410-732-0500 ext. 1030 >> mkaminowitz(a)baltometro.org >> www.baltometro.org >> >> >> >> >> _______________________________________________ >> ctpp-news mailing list >> ctpp-news(a)ryoko.chrispy.net >> http://ryoko.chrispy.net/mailman/listinfo/ctpp-news >>

11 years, 9 months

Re: [CTPP] Adventures in large datasets

by hprawiranata mitcrpc.org

Krishnan, HDF was data format, all climate data is stored on HDF format (I experience writing C code for this data format on SPARC computers long time ago and from HDF I converted to MySQL ! ). CTPP is just a flat data set, txt. Converting into HDF is a different complicated story and on a wrong direction. Linking data tables from many tables has to be done using database engine. MS Access won't cut it. M$ SQL is expensive. Free, Open Source is the only way and fast. TransCAD... I never test the limit. But linking simple tables is ok. Large and many tables ? Better use data modeler software and link to database engine. Hary(ono) Prawiranata Transportation Analyst/Modeler Tri-County Regional Planning Commission 3135 Pine Tree Rd. Ste 2C Lansing, MI 48911 On Thu, Nov 14, 2013 at 9:58 PM, Krishnan Viswanathan <krisviswanathan(a)gmail.com> wrote: > Mara > > Besides SQL server I have the following suggestions: > 1) the ff package in R ( > http://www.bnosac.be/index.php/blog/22-if-you-are-into-large-data-and-work-…) > 2) HDF5 seems like a decent option though I have not used it. Link to rhdf5 > ( http://bioconductor.org/packages/release/bioc/html/rhdf5.html). Also, > SFCTA has some code for getting data into and out of HDF5 ( > https://github.com/sfcta/TAutils/tree/master/hdf5) > 3) I have found TransCAD to be efficient in processing large datasets. > > Hope this helps. > > Krishnan > > I downloaded the Maryland state raw data (the whole enchilada) that Penelope > was good enough to provide me. It came with documentation that clearly > explains what needs to be done but I am being hampered by the sheer size of > the dataset. It's 10 GB and that's without going into joining tables, > transposing them to meet my needs, etc. Even breaking the parts into > different databases it can't be handled in Access. I can fit Part 1 into an > ESRI geodatabase but I don't have the flexibility in linking tables that > Access has. > > > > Does anyone have any suggestions for dealing with large databases? SQL > server is one option. Are there others? > > > > Mara Kaminowitz, GISP > GIS Coordinator > ......................................................................... > Baltimore Metropolitan Council > Offices @ McHenry Row > 1500 Whetstone Way > Suite 300 > Baltimore, MD 21230 > 410-732-0500 ext. 1030 > mkaminowitz(a)baltometro.org > www.baltometro.org > > > > > _______________________________________________ > ctpp-news mailing list > ctpp-news(a)ryoko.chrispy.net > http://ryoko.chrispy.net/mailman/listinfo/ctpp-news > > > _______________________________________________ > ctpp-news mailing list > ctpp-news(a)ryoko.chrispy.net > http://ryoko.chrispy.net/mailman/listinfo/ctpp-news >

11 years, 9 months

[CTPP] Adventures in large datasets

by Mara Kaminowitz

I downloaded the Maryland state raw data (the whole enchilada) that Penelope was good enough to provide me. It came with documentation that clearly explains what needs to be done but I am being hampered by the sheer size of the dataset. It's 10 GB and that's without going into joining tables, transposing them to meet my needs, etc. Even breaking the parts into different databases it can't be handled in Access. I can fit Part 1 into an ESRI geodatabase but I don't have the flexibility in linking tables that Access has. Does anyone have any suggestions for dealing with large databases? SQL server is one option. Are there others? *Mara Kaminowitz, GISP*GIS Coordinator ......................................................................... *Baltimore Metropolitan Council* Offices @ McHenry Row 1500 Whetstone Way Suite 300 Baltimore, MD 21230 410-732-0500 ext. 1030 mkaminowitz(a)baltometro.org www.baltometro.org

11 years, 9 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

CTPP