Wednesday, March 21, 2012
PowerDesigner 16 only uses default printer
Version 16.0.0.3576 EBF5
OS: Windows 7 Professional
Issue: I want to print a multiple page diagram from a Physical Data Model. I open the model, open the diagram, and press Ctrl+A to select all the symbols in the diagram. From File menu, I select "Print Selection". In the "Print Diagram" dialog window, I choose a destination printer, which is not my default printer, choose the paper size and other settings and click "OK". The print job goes to my default printer which doesn't have the paper size I need.
Resolution: Change my default printer to the desired printer for this job and, when done, change my default printer back to normal
Secondary Issue: PowerDesigner does not recognize change to default printer until after it is closed and reopened.
Sybase Inc. Case-Express bug report opened:
Case Id: 11727580 - PowerDesigner only uses default printer
Tuesday, July 12, 2011
Is the relational database doomed?
Have you ever been asked about document-oriented or key value (KV) data stores / databases? Read the article at the link above for a good background on them.
Start with data warehouse or datamart?
But, in the case where a brand new BI solution is being developed, here are my preferences:
- Design the BI solution for the future.
This means planning ahead so that whatever you start with, you are planning for future enterprise data standards, naming conventions, global uniqueness, data quality, and data governance. Do not operate in a business silo. - ROI is important.
While it is difficult to prove a return on investment for a non-tangible product such as BI, most business funders of a BI solution have a niche perspective they want results for quickly. Since building an Enterprise Data Warehouse (EDW) averages about 18 months, focused business data marts (DM) can be completed relatively quickly in six months or less. - Build both DW (to become EDW) and DM.
When starting a new BI solution, don't consider an "either / or" solution. A DM should get its data from the data warehouse (DW), which should be designed for the future EDW. So, bring the master and transactional data from the source-of-record (SOR) database system into a small DW first. This small DW could have only a few subject areas yet still be the beginning of the future EDW. From the small DW, which should be designed as a normalized model in at least 3NF, create the DM that is designed as star (dimensional) model. The advantage of this approach is that the SOR systems only need to populate a single DW (to become future EDW) and all DM created will be sourced from the same source of truth. Thus they will all start with the same data and this will reduce the chances of different metrics being reported from different DM.
Friday, July 1, 2011
Testing data warehouse - Toolbox for IT Groups
Testing data warehouse - Toolbox for IT Groups
My response to the posted question concerning QA testing of Data Warehouse ETL is:
More than just using independently developed SQL statements, I recommend writing separate ETL programs using another programming / scripting language such as Perl to mimic what the ETL programs are supposed to be doing.
Since the same effort will be needed by both the ETL development team and the QA development team, QA cannot be left to the end as, in many projects, a "necessary evil" at the last minute. Both development teams will need to be working independently to, hopefully, achieve the same results.
For example, I had a QA lead project where the ETL programs were written in Pro-C++ moving data from a mainframe DB2 extracted file set to an Oracle UNIX-based enterprise database. Using the same design specifications that the Pro-C developers used, I wrote Perl scripts to create UNIX-based "tables" (file stores) that were to mimic the Oracle target tables.
A complete set of data extract files were created from the DB2 source. Using the same DB2 extract files, the developers ran their programs to load the target tables and I ran mine to load the UNIX files. Then I extracted the data loaded from the Oracle tables and saved the extract to another set of UNIX files in the same format as my extract files. Using UNIX diff command, I compared the two sets of files.
By doing this, I revealed two glaring errors that the developers said would not have been caught using just a small sample set of data extracts:
First, the Pro-C programs were doing batch commits every 2,000 rows, but the program was not resetting the counter in the FOR loop. Thus, only the first 2,000 rows were getting written to the Oracle tables. If the source-to-target mapping predicted a million rows, the resulting 2,000 rows was a big problem!
After this was fixed, the next glaring error my Perl scripts caught was that the variable array in Pro-C was not being cleared properly between rows. Therefore, some fields from the previous row were not cleared and were duplicated for subsequent rows. Thus some customers records that did not supply certain fields, say an email address, got the data in those fields from the previous customer record.
Since these programs were written using the same design specifications, but independently, one of the checks was to give the code used to each other, so I checked their Pro-C code and they checked my Perl code to see where the problem was.
Simply sending a few hundred rows of mocked-up test data to the Oracle target and checking that it matched expectations was not adequate testing and would not have revealed these serious bugs.
Thursday, June 30, 2011
How to handle Legacy Key of entity while migrating data into MDM Hub? (Linked In)
My contribution to this discussion:
Steve has made some very good suggestions (best practices). I'd like to give my support to:
- Keep the source (You said "legacy"? Does this mean the source system is being retired?) system's Primary Keys (PK) in the MDM.
- Create a PK in the MDM using a surrogate value "surrogate key".
- No new rows or attribute values are to be created in the MDM that did not originate from the source system; that violates the principle of MDM.
- If an additional column in the MDM table for storing the source system's PK is not desirable, then you can, as Steve suggests, use another cross-reference table to store the source-to-MDM PK relationships. This can get messy if you need to do this for multiple sources or if the source system reuses its PK values - I have seen this happen.
Also, "downstream" components using the MDM data should only use whatever attributes they would use if getting their data directly from the source system. If the source system is using a surrogate PK value, then it is unlikely that users would know what it is. They should be using some "natural key" attribute or attributes that uniquely identify the rows. If they do know and use the source's PK values, that is the attribute they should continue using; the MDM's PK (if a surrogate value) should not be known to them. Since column and table names are sure to be different in the MDM than they are in the source, create views through which the users can access data using the source systems' table and column names.
For a dissertation on "surrogate keys", you might want to read http://itpro420.blogspot.com/2011/06/what-is-difference-between-surrogate.html.
Tuesday, June 28, 2011
Type 4 Slowly Changing Dimensions Design - Toolbox for IT Groups
I was just wondering in a type 4 SCD
My response to this question is:
First of all, a good understanding of what a "surrogate key" really is is needed. For that, see my blog site at http://itpro420.blogspot.com/2011/06/what-is-difference-between-surrogate.html.
As I state in my blog, there really is no such thing as a "surrogate key"; the term is used to describe a Primary Key (PK) that uses a surrogate value.
If you follow C. J. Date's rules, every table would have a PK using a surrogate value, ergo a "surrogate key". But each table would also have a "natural key", or a column or group of columns that uniquely identify a row (enforced by a unique index).
If the source database system uses a "surrogate key" (PK in the source "current database") that is brought into the data warehouse (DW - a "temporal database"), that column's value will never change. However, the row in the DW should also have a PK that uses a surrogate value. In order to keep track of changes to specific columns in that row, audit columns must be used such as "start date", "end date", "current".
Read a good, IMHO, definition of "surrogate key" at http://en.wikipedia.org/wiki/Surrogate_key.
Now, an understanding of Slowly Changing Dimensions is needed. Since you mentioned Type 4 (SCD4), let me explain that this is nothing more than a Type 3 (SCD3) kept in a separate table. Therefore, SCD4 requires a "current dimension table" that contains only currently valid records and separate "historical dimension table" that contains the history of changes; SCD3 does this in one table. I have rarely seen SCD4 used because it requires more database space, management, and views to use by users to find the value that is either current or in use at a previous time. But, in either case, the DW PK that uses a surrogate value and the source's PK using a surrogate value will never change.
One question: in the IT field do you receive a job opportunity because of who you know or what you know? | LinkedIn
My response to this question:
Both "who" and "what" are important; it is not an "either or" situation. A person (friend, colleague, network contact, ...) - the "who" - can give you a "heads up" or maybe get your resume to a hiring manager, but it seems that most large companies (where the jobs are) now use recruiting firms to screen applicants. But if you don't meet the job's requirements - the "what" -, that won't get you the job and it will make your "who" look bad. Entry-level jobs in the US are few and far between. It seems most jobs (contract or FTE) are looking for mid-level experience (around 5 years) in specific skill sets. Certifications without work experience will only be better for you if you are competing against someone without either certifications or work experience - you have an advantage for an entry-level job. To get work experience, you can try to do pro bono work for charities, a friend's business, small businesses that need some small help; you can also try temp agencies, but they may require work experienced, too. This pro bono and/or temp work could lead to long-term contract or FTE recommendations. Good luck.